Draft implementation of generators.yaml spec

[SuprDewd]

This is a first attempt at an implementation of the generators.yaml draft spec described in https://github.com/Kattis/problem-package-format/pull/2

It adds the generatedata command which has the following synopsis:

usage: generatedata [-h] [-g] [-c] [-C] [-n] [-j PARALLELISM] [-b]
                    [-l LOG_LEVEL] [-e]
                    [--max_additional_info MAX_ADDITIONAL_INFO]
                    problemdir [problemdir ...]

Generate test data for a problem package in the Kattis problem format.

positional arguments:
  problemdir

optional arguments:
  -h, --help            show this help message and exit
  -g, --generate        generate test data
  -c, --clean           clean up generated files
  -C, --clean_all       clean up generated and unrecognized files
  -n, --dry_run         don't actually do anything
  -j PARALLELISM, --parallelism PARALLELISM
                        level of parallelism
  -b, --bail_on_error   bail verification on first error
  -l LOG_LEVEL, --log_level LOG_LEVEL
                        set log level (debug, info, warning, error, critical)
  -e, --werror          consider warnings as errors
  --max_additional_info MAX_ADDITIONAL_INFO
                        maximum number of lines of additional info (e.g.
                        compiler output or validator feedback) to display
                        about an error (set to 0 to disable additional info)

The verifyproblem command is also augmented to validate the corresponding generators.yaml file.

At the moment it does not support testgroup includes, and probably won't until their semantics have been fully decided. Other notable features that are also missing:

• First-class support for .interaction files
• Keep track of how files were generated to determine what needs to be regenerated (in progress)

[pehrsoderman]

@SuprDewd Can you rebase this, so we can get it moving forward?

[SuprDewd]

@pehrsoderman I've finished rebasing.

@thorehusfeldt I added support for testdata.yaml. Do you want to assist with the review by taking this for a spin?

[RagnarGrootKoerkamp]

@SuprDewd What exactly does testdata.yaml support mean in this case? Just allowing them to exist in the data directory? Or are you actually generating them, as I did here https://github.com/RagnarGrootKoerkamp/BAPCtools/pull/108#issuecomment-826823499.

[RagnarGrootKoerkamp]

If you'd like I can also go over the full code, but someone more familiar with problemtools side should probably also do this.

[SuprDewd]

Can we get this merged? This has been tested at three NWERCs without issues.

[RagnarGrootKoerkamp]

Maybe we should also merge the spec itself:) One thing I was thinking of that may be nice would be a way to directly write small testcases in the generators.yaml, avoiding the need for an stsout.py generator or separate files.