search

Kattis / problem-package-format

Kattis problem package format specification
https://www.kattis.com/problem-package-format
9 stars 14 forks source link

Drop languages setting in problem.yaml #123

Closed niemela closed 1 year ago

niemela commented 1 year ago

Should we drop the languages setting the problem.yaml?

I don't really see a use case for this, it's been used 0 times on Kattis.

Can anybody else see a need for this?

Tagl commented 1 year ago

I don't think that's a fair statistic to use since problemtools didn't support it before yesterday (after my PR got merged). So nobody would use the keyword that just added an error. There are problems on Kattis which technically should have languages set to cpp. Some are from @jsannemo 's course.

I definitely see a use for it. Whether it is implemented using include, the languages keyword or some other way is not the most important thing to me, but being able to limit the set of languages matters to me.

In an educational setting at least, I don't want to give the option of using another language if it will always result in Wrong Answer or Run Time Error no matter what they do.

On Tue, Sep 19, 2023, 02:09 Fredrik Niemelä @.***> wrote:

Should we drop the languages setting the problem.yaml?

I don't really see a use case for this, it's been used 0 times on Kattis.

Can anybody else see a need for this?

— Reply to this email directly, view it on GitHub https://github.com/Kattis/problem-package-format/issues/123, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2ZBKU3T7INCB5EVJ44TKLX3D5E5ANCNFSM6AAAAAA45SJRGE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

pehrsoderman commented 1 year ago

I agree with Tagl here. It makes a lot of sense to be able to restrict some problems to specific languages.

Tagl commented 1 year ago

These are some I know from the top of my head that should have cpp as the only language:

niemela commented 1 year ago

Ok, all good points. Let me try a bit more though...

For all of the problems listed, and all of the problems I can think of, that should be limited to some set of languages, this is true because they use included code in a way where it's not possible to solve the problem without the included code, and included code is not provided for all languages.

Is there any other reason why a problem should be restricted?

If this is the only (or almost only) way that this can happen, then I think it makes sense to connect the two. I.e. instead of explicitly setting a languages property, we could say that a problem is restricted iff there are some included code specified, and then it is restricted to exactly those languages that are specified.

Some examples:

  1. If the problem requires some included code to work (typically a driver), one would set include for the languages one want to support, and the language restriction would be implied correctly.
  2. If the problem does not require included code to work (typically this means that we want to add some helper functions to some languages, but these are not strictly needed), one would set include for those and and then also an (possibly empty) all (or default see #114).
  3. If the problems needs to be restricted to languages but there is no included code, then one would simply set empty include for those languages.

The 3rd option is a bit hacky, but I have never seen the need for that, or heard of a use case for it.

What do you think? Would you like this way of removing the languages property from problem.yaml but keeping the ability to restrict tom some languages?

Tagl commented 1 year ago

Well I think some might want to restrict language anyway, even without a driver or some provided library. Could be in an educational setting, or a weird programming contest where one problem should be solved in Befunge or some esoteric language. It could simply be part of the task to use a specific language or only be allowed to submit text or something else. Many teachers would want to restrict the package to a single language, at least that is my experience at Reykjavik University, though I disagree with doing that for simply the reason that they teach that language in a course. Now they could maybe do this in the judge system, but only if the judge system supports that and they have access to changing it. Alternatively they can set it in the problem and the judge system should respect it.

I think it is fair to use include since that will be 99% of cases where you do this. Not sure I have any other suggestions for alternative methods. Maybe a local languages.yaml for the problem could also work to provide even more flexibility, but that might be a bit extreme.

niemela commented 1 year ago

Well I think some might want to restrict language anyway, even without a driver or some provided library.

Could be in an educational setting, or a weird programming contest where one problem should be solved in Befunge or some esoteric language.

I would argue that that is not a property of the problem though. I definitely would not want "problem X, but you can only use COBOL" to require you to create a new problem package (even if the process would admittedly be trivial), rather this is something you should configure in your judging/teaching/contest system. And, in fact, that is how we currently do it. We don't specify that problems for most ICPC contests only allow C, C++, Python, Java, Kotlin, we just make sure to only make those languages available in the system.

My point is that language should mean you couldn't (or at least shouldn't) ever be able to solve this problem in any other language.

It could simply be part of the task to use a specific language or only be allowed to submit text or something else.

Same argument as above.

Many teachers would want to restrict the package to a single language, at least that is my experience at Reykjavik University, though I disagree with doing that for simply the reason that they teach that language in a course.

I also strongly disagree with this from a pedagogical/philosophical point of view, but that's not the argument agains. The problem package shouldn't be that opinionated :).

Now they could maybe do this in the judge system,

Yes.

but only if the judge system supports that and they have access to changing it. Alternatively they can set it in the problem and the judge system should respect it.

How likely is it that a judging system supports and respects language restrictions, but does not offer it as a setting? (That's not a rhetorical question, I'm not claiming that's it's obviously very unlikely)

I think it is fair to use include since that will be 99% of cases where you do this. Not sure I have any other suggestions for alternative methods. Maybe a local languages.yaml for the problem could also work to provide even more flexibility, but that might be a bit extreme.

What would languages.yaml contain other than the list of allowed languages?

But, yeah, if anything I want to simplify this, not "provide even more flexibility" :).

RagnarGrootKoerkamp commented 1 year ago

I don't have a stake in this, but I do agree with Fredrik's argument:

Once possible limitation with the implicit inferring via includes:

niemela commented 1 year ago

[...] Once possible limitation with the implicit inferring via includes:

  • I could see a situation where you may want to provide an include for python (eg a balanced binary tree implementation), but nothing for other languages. In that case you'd still want to allow all languages, but only have an include directory for python.

Yes, and the solution is that in that case you should add an include directory for default (that maybe should include an README or something, since empty directories don't work so great with git).

Tagl commented 1 year ago

How likely is it that a judging system supports and respects language restrictions, but does not offer it as a setting? (That's not a rhetorical question, I'm not claiming that's it's obviously very unlikely)

It may support that, but on a group level usually, like contest or assignment with multiple problems. My gradescope integration supports only the languages keyword for restricting languages, because each assignment is a single problem so I didn't see reason to restrict it the global configuration. But all those problems also use include anyway.

I think we agree that problems shouldn't make these restrictions for no reason, however it will always be possible to do so since the driver can just run their program without changes for a single programming language. I was just playing devil's advocate because I know teachers that want to do this.

If nobody opposes tying languages with include then I think that is our best option.

niemela commented 1 year ago

@pehrsoderman ?

niemela commented 1 year ago

I discussed this with @pehrsoderman. I think we still need the languages setting for some use cases. The best example would be a problem where the problem statement is written in such a way that it explicitly instructs the solution to be written in a given language.

Closing this.