The suite should explicitly document its backwards compatibility policy

[Julian]

This suite is generally not versioned (despite such things having been requested in the past).

Regardless though, it is important to define exactly what sorts of backwards compatibility we offer. E.g. -- do we guarantee that files will stay in the same location? That tests will not change descriptions? That the directory structure won't change?

Some of these things have changed recently, so to help those who use the suite, we should explicitly note which of the above (and others) we guarantee.

Possibly here we should also therefore change the structure of the repo into one that clearly delineates between an authoring format and an output format -- e.g., from an authoring standpoint, it's potentially annoying to have to replicate tests across drafts, so perhaps tests should by default be replicated, unless explicitly restricted. But from an output standpoint, it's likely more convenient to fully denormalize the output, so that users of the suite can treat each version completely independently.

[NickNeck]

Hello,

here is just a little idea to start the discussion.

The test suite should provide a versioning like the semantic versioning. In this case, we can guarantee backward compatibility in some gradations. I think the original specification of "Semantic Versioning" doe not fit to 100% to the test suite, so we have to define our own meaning to the mayor, minor, and patch labels.

Things that can be guaranteed for the project, even the obvious:

• All tests are located at /tests.
• All test suites are direct subdirectories of /tests and have the format corresponding to the regular expression /draft\d+/.
• The /tests/draftXY/ subdirectories contain json test files or other subdirectories in an indefinite number. The subpath to a test file without the extension .json will be called group.
• All test files are valid against the schema test-schema.json
• In a test file are all values for description in the same array unique.
• The object that contains the tests will be called case.

With this rules a particular test can be addressed by:

suite: "draft4"
group: "items"
case: "an array of schemas for items"
test: "wrong types"

Furthermore, a release file should be created, that contains all tests in a single file. This file would look like this (simplified with just one test):

{
  "version": "1.2.3",
  "description": "JSON Schema Test Suite",
  "suites": [
    {
      "description": "draft4",
      "groups": [
        {
          "description": "items",
          "cases": [
            {
            "description": "an array of schemas for items",
              "schema": {
                "items": [
                  {"type": "integer"},
                  {"type": "string"}
                ]
              },
              "tests": [
                {
                  "description": "correct types",
                  "data": [ 1, "foo" ],
                  "valid": true
                }
              ]
            }
          ] 
        }  
      ]
    }  
  ]
}

This file would be placed at rel and named after his version /rel/json-schema-test-suite.1.2.3.json. Every version has a related git tag. Additionally, there would be files with a reference to the latest version.

/rel/json-schema-test-suite.1.0.0.json
/rel/json-schema-test-suite.1.0.1.json
/rel/json-schema-test-suite.1.0.2.json
/rel/json-schema-test-suite.1.1.0.json
/rel/json-schema-test-suite.1.1.1.json
/rel/json-schema-test-suite.2.0.0.json
/rel/json-schema-test-suite.1.0.latest.json  # same as 1.0.2
/rel/json-schema-test-suite.1.1.latest.json  # same as 1.1.1
/rel/json-schema-test-suite.1.latest.json    # same as 1.1.1
/rel/json-schema-test-suite.2.latest.json    # same as 2.0.0
/rel/json-schema-test-suite.latest.json      # same as 2.0.0

And one file with the name json-schema-test-suite-dev.json with all tests in the current master branch and a Version like 2.0.0-(14)-a6ljbel7

Finally, the question remains when which part of the version will be increased.

• Increasing the patch part of the version when several changes for the tests and/or the cases are available.
• Increasing the minor part of the version when changes for the groups are available.
• Increating the mayor part of the version for all other changes.

[Julian]

Thanks for kicking that off, that's exactly the kind of thing I want to document!

Will have to leave some detailed comments as soon as I can review (others of course are welcome to).

[Julian]

Two random things --

First, directed at myself in 2018 -- let's keep this ticket to just be about establishing the first baseline backwards compatibility policy (which we'll then need to adhere to for at least 6 months as downstream authors become aware of it).

So no changing structure yet, not in a way downstream authors will notice, so we might as well leave that topic out for now and revisit it in its own ticket.

Second random thing -- probably bears thinking about #244 in this context -- e.g., where will metaschema tests live? In a file called metaschema.json? In a directory called metaschema? If the answer is the latter we may indeed have another sub-directory inside a tests folder, so we should cover that.

Oh, I guess third random thing is a general recommendation to downstream users on whether they should write their suite loaders as "opt in by default" or "opt out by default" if they encounter a new file.