search

Kattis / problem-package-format

Kattis problem package format specification
https://www.kattis.com/problem-package-format
9 stars 14 forks source link

make data for testcases (again) #338

Open mzuenni opened 2 months ago

mzuenni commented 2 months ago

With the last changes data/samples/downloads/ was introduced, which is useful to overwrite testcases for the sample.zip. However, it was decided that also non testcase related stuff should be placed there. But I think it is better if files in data should correspond to testcases .

For clarification /data/sample/<xxx>.in and /data/sample/<xxx>.ans both belong to the testcase /sample/<xxx> and files like /data/sample/downloads/<xxx> and /data/sample/statement/<xxx> also belong to that testcase. But a file like /data/sample/downloads/testing_tool.py does not belong to any testcases. Unfortunately this has the following side effects:

And this does not only happen for testing tools but anything that does not match a testcases. Therefore I would propose to have attachments for stuff that does not belong to a testcase and make /data only for files to belong to a specific testcases.

RagnarGrootKoerkamp commented 2 months ago

More generally, we should consider what exactly we intend to be the difference between /attachements and /data/sample/download. Why do we present downloads in two places anyway in the UI? If we merge those from a user point of view, we can simply put the testing tool in attachments again.

Say you have some dictionary data attachement. Why should that not be included in the 'sample download' directly. Also if you run a contest and provide a zip with all samples, it does make sense for attachments to also be included right there, right?

If we do want to keep the attachments vs samples download, we could also consider a new top level /testing_tool directory, or simply /downloads.

(Indeed, I agree that the current data/sample/{.,statement,download} gives a lot of flexibility, making it hard to check that things are sane.)

evouga commented 1 month ago

Has this issue been resolved? The current draft spec does have attachments.

evouga commented 1 month ago

We do say

If you want to provide files related to interactive problems (such as testing tools or input files), you can use data/sample/download

and I agree that attachments is a better place for testing tools.

RagnarGrootKoerkamp commented 1 month ago

No this is still open. We have have overshot a bit when we were writing this up in Stockholm. I'm not on top of this atm though.

@mzuenni: Is all you are proposing that the testing tool moving out of data/samples, and the rest there stays the same? Or would you prefer to also move 'testing-tool-only inputs' back to attachments/?

mzuenni commented 1 month ago

I am not sure what you mean with "testing-tool-only inputs". I would say that everything that does not belong to one specific testcase does not belong in data. (in bapctools generator.yaml terms: everything that is not numbered should not be in data)

niemela commented 1 month ago

I very much agree with the "there should only be actual test data and metadata directly related to it" in the data/ directory. Thus, testing tools should be in (what's now called) attachments/.

Why do we present downloads in two places anyway in the UI?

That is not specified by the format, so "we" are not doing that. How attachments/ is really up to the system. For an online judge such as Kattis, it makes a lot of sense to offer them as downloads, but in a contest setting (with an on-site judge) it could be better to make those files available on the contestants machines. That sample input/output is being available as files at all, is not required either, it's just nice, and I think all of us that have web-based systems do it as downloads.

Another bad thing with the current naming of things is that "attachments" and "downloads" are clearly used as synonyms. That's generally bad design.

That said, the reason we called attachments/ that and not downloads/ is that "attachment" is what it is (i.e. the semantics), whereas "download" is how it (could) be delivered (i.e. the mechanics). As pointed out above, there are other reasonable ways to deliver "attachments" then downloading them. OTOH, for /data/sample/downloads I don't think "attachments" works, which is why we ended up with downloads there, but we don't (and shouldn't) require them to actually be "downloadable", at most that they should be made available.