search

RagnarGrootKoerkamp / BAPCtools

Tools for developing ICPC-style programming contest problems.
GNU General Public License v3.0
48 stars 18 forks source link

Testdata semantics (validation flags) and inheritance #349

Closed thorehusfeldt closed 4 months ago

thorehusfeldt commented 4 months ago

This is a necessary step on the roadmap to supporting type: scoring and implementing the expecations framework.

testdata.yaml is a file that can exist in the subdirectories of data and modifies input and output validators as well as the rules for scoring aggregation. Two observations:

  1. testdata.yaml plays little to no role in BAPC traditions, and does not have a lot of attention from BAPCtools
  2. The specification has changed quite a bit, trimming down the scope, changing the syntax for targeting individual input validators, and deciding the inheritance rules

testdata.yaml affect a large part of BACPtools, including generation, validation, and running.

Roadmap:

  1. construct a pathological problem for test/problems that uses all kinds of weird details (inheriting for two different input validators, for instance.) This can be a pass/fail-problem. See what already works.
  2. Update generate.py, this may be minimal – it’s just a file to be written. However, the CUE schema needs to have a working #testdata_settings, and the JSON schema is probably also either trivial or outdated.
  3. Update validate.py. Input validation (I think) works; output and in particular answer validation is a long shot.
  4. Update run.py and probably testcase.py.

The main issue I’m worried about is that answer validators ought to be run with the flags pertaining to output validation of the given test case. But BAPCtools stores flags per validators. This is great for input validation, but not for answer validation. So I assume this needs a more general data structure.

Once all of that it is done, we can proceed to problems with type: scoring (which also need to access testdata.yaml, but the inheritance rules are simpler.)

RagnarGrootKoerkamp commented 4 months ago

Do you need specific input from me here?

Generally I haven't used this so don't have many opinions on how testdata.yaml behaves for users.

From the code side, I see that class Testcase already contains self.testdata_yaml so that shouldn't be a problem but indeed we should write out in detail what is the expected behaviour of input/answer/output validators with these additional flags present.

thorehusfeldt commented 4 months ago

I’ll call for help when I need. I assigned myself to this task for now, just to signal that nobody else should spend clock cycles on it.

thorehusfeldt commented 4 months ago

Solved by #355