search

Kattis / problemtools

Tools to manage problem packages using the Kattis problem package format.
MIT License
101 stars 70 forks source link

Update default grader with first_error, worst_error, ignore_sample, prefer_accepted #121

Closed simonlindholm closed 5 years ago

simonlindholm commented 5 years ago

Recently added to the spec after some discussion with @niemela. first_error is rarely useful, but can be good for avoiding timing leaks and for analytics in judges that don't do lower-priority judging of unnecessary testcases. worst_error is a better name for no_errors. ignore_sample is useful for scoring problems (unless secret/ has a grader that always outputs Accepted (0), as we're doing with PO; a Rejected status would help with that).

simonlindholm commented 5 years ago

Now updated with prefer_accepted and some validation in verifyproblem.

austrin commented 5 years ago

Is it still WIP or is it ready to merge?

What exactly is the use case of prefer_accepted?

simonlindholm commented 5 years ago

Ready to merge! prefer_accepted is for subgroups, where the secret/ directory should get Accepted if any subgroup does, but otherwise use something like first_error or worst_error.

austrin commented 5 years ago

While flying I was thinking that prefer_accepted is maybe not an optimal name. Not sure what would be, but wouldn't something like accept_if_any_accepted or one_accepted_sufficient be a bit better (more descriptive)?

ghamerly commented 5 years ago

While flying I was thinking that prefer_accepted is maybe not an optimal name. Not sure what would be, but wouldn't something like accept_if_any_accepted or one_accepted_sufficient be a bit better (more descriptive)?

at_least_one_accepted? one_or_more_accepted? any_accepted?

austrin commented 5 years ago

@ghamerly I think your suggestions are not great -- the point of the flag is to say that "accept even if there is just a single AC sub-result". So e.g. at_least_one_accepted sends the wrong message because you need at least one accepted to get AC both with and without the flag.

I think my preference is the very explicit one_accepted_sufficient. But I guess prefer_accepted is fine. @simonlindholm what do you think?

simonlindholm commented 5 years ago

I think I'd be happy with accept_if_any_accepted.

niemela commented 5 years ago

I'd accept (pun intended) accept_if_any_accepted, but I'd prefer (yes...) prefer_accepted.

simonlindholm commented 5 years ago

Renamed to accept_if_any_accepted (so requires another spec update). Let's try to get this merged.

niemela commented 5 years ago

Spec is updated. Go ahead and merge!