search

MiniZinc / mzn-bench

A framework to performing benchmark testing on MiniZinc models, solvers, and/or the compiler itself.
https://www.minizinc.org
5 stars 5 forks source link

Skip checks on ERROR statuses #9

Closed hbierlee closed 3 years ago

hbierlee commented 3 years ago

The method field might be missing for solutions with an ERROR status, which this change handles by just skipping the test case for errors. However, this raises the question how to see an ERROR status, as you can argue whether it passes, skips or fals the test. An error can be a bug (in which case it might be good if it the test case is failed) or it can be the memory running out (which is not really a failure, like an UNKNOWN is not if the timelimit runs out). Let me know what would be a good procedure for this :+1:

cyderize commented 3 years ago

I think this makes sense since an ERROR status never contradicts any other statuses - I don't think you can say that having an error was 'incorrect' - it definitely had an error - so I would say that errors should either pass or be skipped. Skipping is probably the clearest way to make the user aware of what's happening.