Comparing output to files (as opposed to stdout)

[niemela]

Let's say you wanted to define the following problem (and I would argue that something like this would be useful in an educational setting):

Read several numbers from numbers.txt, one number per line. Write their sum to sum.txt and their product to product.txt.

We can define this problem in the current version. You would put the numbers.txt-files as <testcase>.files/numbers.txt. For validating you would have to use a custom validator. You would put the sum and product in <testcase>.ans and the validator would get .ans and usual, and would have to know how to parse it and what should go in what file. This is not very different from any validator that gets some kind of secret from the .ans file.

The implication is that you will always have to use a custom validator when you want the submission to write output to a file.

If we don't want this we could introduce something like an .ansfiles directory. The default validator would then for each file in .ansfiles verify that this exists and is equal in the working directory of the submission after running it.

My sense is that requiring a custom validator whenever you use file output is acceptable, and that adding something like .ansfiles is more complication than it's worth. So, I suggest that we don't do anything. I think @Tagl also agrees?

Thoughts?

---

A related question is allowing writing at all.

I would argue that we do want the kind of problems suggested above, and then we have to allow writing to file (that is in fact the point). That said, in the normal stdin/stdout case it would be nice if the system is a at least allowed to forbid writing to files. Should there be a setting that you are required (as a problem setter) to use to explicitly allow file writes? This would also allow a system to know at install time that a problem will require writing to file, and thus be able to refuse the problem at install time, rather than failing at run time.

[Tagl]

I think @Tagl also agrees?

I do agree.

Should there be a setting that you are required (as a problem setter) to use to explicitly allow file writes?

I think yes, boolean is probably enough. Potentially you would want to have an additional setting that says writing new files is allowed but editing submitted files is not, or even 4 settings if you consider included files too. I think that's not needed though and results in too much complexity without a proper use case in mind.

[eldering]

Thinking about our setup in DOMjudge, if allowing submissions to write to files, I'd prefer if the problem would then somehow specify which files and/or directories, or to have a default directory to write to, like the feedback_dir for validators. That way, I can shield off anything else by not giving the submission write permission to any file or directory than those explicitly allowed.