Symlinks for reuse of input files with different output validator flags

[Tagl]

https://github.com/Kattis/problemtools/blob/92dcbf3fab45ed4c13438511df98ec82fec512f9/problemtools/verifyproblem.py#L212C16-L212C16

Should the above not be a warning, rather than an error? I don't see why it would be an error. Using symlinks is a nice way of reusing the input/answer files. For example, you can use the same test cases in two different groups. In one of them validation is case insensitive and in the other a stricter case sensitive validation is applied.

[Tagl]

Results from symlinked tests are also reused even though output validator flags are different. That is definitely incorrect behaviour.

[niemela]

Agreed, this is not what we want.

@austrin @ghamerly Do you see any reason why it would be dangerous to fix this?

[ghamerly]

I cannot think of a reason why this should be prohibited. This is a question for @simonlindholm who wrote the code originally, I think.

[gkreitz]

Looking at the code, it seems to me that these symlink checks were added for some quite particular use case, where one also wanted the optimization of re-using results. So, it makes some sense that someone who wanted that particular use cases added a guard (the error) to catch when results were re-used dangerously.

We should definitely fix this. I'm somewhat inclined to simply remove the optimization of re-using results when test cases are identical and I/O-files are symlink:ed - it feels like a very corner-case scenario for the code complexity. If we do want to keep it, we should change what are now error messages to instead simply disable the re-use of results.

[simonlindholm]

It is very much not a corner case scenario; it would at least double judging times for scoring problems with test groups, where we more often than not include earlier test groups in later ones.