search

Kattis / problemtools

Tools to manage problem packages using the Kattis problem package format.
MIT License
105 stars 72 forks source link

Default output validation fails for `inf` and `nan` answer #193

Open RagnarGrootKoerkamp opened 3 years ago

RagnarGrootKoerkamp commented 3 years ago

Admittedly this is rather pedantic, so feel free to just close, but in case you are interested:

I just added some tests to the BAPCtools default output validator to handle inf, -inf, and nan as possible answers to testcases in the presence of float_tolerance 0.5.

Some cases that problemtools currently fails to handle: answer output expected actual
inf inf ac wa
inf INF ac wa
inf -inf wa ac
inf 0 wa ac
-inf -1 wa ac
nan nan ac wa
nan NaN ac wa

(You could argue what relative error means in the context of infinite values, but some of these are clearly wrong.)

To run some tests, clone the BAPCtools repository and change line 50 to:

validator = validate.OutputValidator(p, Path('/path/to/problemtools/support/default_validator/default_validator.cc'))

and run pytest -k default_output from the repository root.

CC @ludopulles who found this