Suggestion: split edge case tests from normal tests

[christiaangoossens]

I noticed that there were some tests that used edge cases to test normal flow, such as:

• Sending startIndex as 0 (in UserCrudTests.java and GroupCrudTests.java), while the spec says it's a 1-indexed array. This tests the edge case of A value less than 1 SHALL be interpreted as 1., but will fail on some servers, such as https://github.com/scimmyjs/scimmy/issues/42
• Sending count as 0 (in SearchTests.java), which tets the edge case of A value of "0" indicates that no resource results are to be returned except for "totalResults". (also https://github.com/scimmyjs/scimmy/issues/42)
• Sending a number of fixed properties in FakeData.java even if the server does not advertise them, such as phoneNumbers and name.honorificPrefix. The SCIM spec details that you should ignore parameters that you do not define, which makes it great for an edge case test, but maybe we should discover available properties (just like you currently do with custom schemas) or send a more limited required set. (See https://github.com/scimmyjs/scimmy/issues/41)

Splitting these tests could help users identify that their SCIM server is working on the basics (minimal or standards requests pass) but that it lacks specific edge cases (negative numbers, 0, undefined properties).

What do you think? I was grateful for the use of these edge cases, as it allowed me to find issues in the server, but it was a bit confusing why it failed and I had to reference the container logs quite a few times.

[suvera]

@christiaangoossens

This tool is aimed to test overall Compliance of the implementation. Yea I agree we should follow the Happy Path in general, as most of the scim implementations do that.

for #1: I think we have to change that to 1 in this tool. for #2: scim implementation has to fix. for #3: It's a limitation in this tool. I agree this tool should read the properties advertized by the server and prepare data accordingly.