Move generators JSON-schema from schemastore to this repo

[thorehusfeldt]

Currently, there exists at https://github.com/SchemaStore/schemastore a JSON schema that validates generators.yaml automatically from editors that support this.

• The JSON schema: https://github.com/SchemaStore/schemastore/blob/master/src/schemas/json/problem_package_generators.json
• Positive tests: https://github.com/SchemaStore/schemastore/tree/master/src/test/problem_package_generators
• Negative tests: https://github.com/SchemaStore/schemastore/tree/master/src/negative_test/problem_package_generators

This schema should instead be hosted directly in the BAPCtools repo, possibly at support/schemas.

To this end,

1. Follow the steps under https://github.com/SchemaStore/schemastore/blob/master/CONTRIBUTING.md#how-to-move-a-json-schema-from-schemastore-to-somewhere-thats-self-hosted
2. Move the .json file to support/schemas and the content of the two test directories to test/, in fact maybe to test/generator_yaml.

Questions:

Q1: should we finally introduce some sort of versioning, so the CUE, YAML, JSON and prose specifications of generators.yaml can refer to each other? The schemastore expects our schemas to have version numbers; see, e.g., the format for referring to our locally hosted version: https://github.com/SchemaStore/schemastore/pull/2421/files . Thus, some kind of decision needs to be made. I unanimously decided that the schema currently at schemastore represents version 0.9 of the generators framework. Possible names for the post-migration version could be 0.9.1 or 1.0.

Q2: schemastore validates the JSON schema against the test using a javascript validator. We have no such infrastructure. Do we want to automate this?

[thorehusfeldt]

More TODO: the page https://github.com/RagnarGrootKoerkamp/BAPCtools/blob/master/doc/generators.md contains two dead links (“here”) to schemata.

[RagnarGrootKoerkamp]

Q1. I think I'm OK with making what we have version 1. I don't think we have any planned open feature requests for generators.yaml syntax at the moment.

Q2. If it's easy, sure we can add some test infra for this. But I'd also be OK to just run this manually whenever the schema changes, as long as we have some clear instructions on how to run it manually. (But by that point, it's probably easy to copy that into CI as well.)

[mpsijm]

Q1: echoing the conclusion from our Slack discussion: let's use yyyy-mm version numbers, which is consistent with the problem package format 🙂

[thorehusfeldt]

I submitted a pull request to the schemastore.

https://github.com/SchemaStore/schemastore/pull/4207

[thorehusfeldt]

Done.

We can either close this issue or keep it open because of the following:

TODO Reestablish testing of the JSON schemata, now use the already-existing (in CI) cue framework.

[thorehusfeldt]

I believe we want to do something like this to validate against the cue schema. (The proposal below contains some suggestion for where to put various files.)

valid_generators_yaml=( "../docs" "../test/problems" "../test/generators_yaml/valid" )
invalid_generators_yaml=( "../test/generators_yaml/invalid" )

any_failed=0

for dir in "${valid_generators_yaml[@]}"; do
    for file in $(find "$dir" -type f -name '*generators.yaml'); do
        echo "Running 'cue vet' on $file..."
        cue vet $file ../support/schemas/*.cue -d '#Generators'

        if [ $? -eq 0 ]; then
            echo "Successfully vetted $file"
        else
            echo "Failed to vet $file" >&2
            any_failed=1 
        fi
    done
done

for dir in "${invalid_generators_yaml[@]}"; do
    for file in $(find "$dir" -type f -name '*generators.yaml'); do
        echo "Running 'cue vet' on $file..."
        cue vet $file ../support/schemas/*.cue -d '#Generators'

        if [ $? -eq 0 ]; then
            echo "Falsely vetted invalid  $file"
            any_failed=1 
        else
            echo "Correctly refused to vet $file" >&2
        fi
    done
done

if [ $any_failed -ne 0 ]; then
    echo "One or more cue vet commands failed." >&2
    exit 1
else
    echo "All cue vet commands completed successfully."
    exit 0
fi

Then we want to also verify against the JSON schema. We can do

cue import generators_yaml_schema.json

which generates generators_yaml_schema.cue and we can then vet against that schema, possibly inside the above for-loops.

We may need to do that inside a bespoke temporary directory so as to prevent the automatically generated generators_yaml_schema.cue to get in the way of the handcrafted .cue schemas (in the CUE package problemformat).

[RagnarGrootKoerkamp]

Yes, this sounds quite reasonable. For a temporary directory we can use mktemp -d. (We could move it to python and into bt but I don't think that's really needed.)