search

DOMjudge / domjudge

DOMjudge programming contest jury system
https://www.domjudge.org
GNU General Public License v2.0
732 stars 256 forks source link

Detect solutions with the same wrong outputs #185

Closed hex539 closed 5 years ago

hex539 commented 8 years ago

It would be useful to be able to group wrong submissions together not just by the test case they failed on, but by what the wrong output was.

For example, a problem with "output 'one' if X < 2, X otherwise" might get a lot of identical wrong submissions for X=1. It's useful to be able to find these.

I'm happy to write something up for this if it's useful.

eldering commented 8 years ago

Hi Robin,

This does remind me of a feature I saw proposed (but not implemented as far as I know) for PC^2: to calculate hashes of team outputs to possibly detect if many teams give the same "wrong" answer, say in case the jury made a mistake.

I'm just wondering how to present this in the jury interface. It can be useful at times, but I think it shouldn't clutter the interface too much. Did you already have something in mind?

On 29-11-15 10:22, Robin Lee wrote:

It would be useful to be able to group wrong submissions together not just by the test case they failed on, but by what the wrong output was.

For example, a problem with "output 'one' if X < 2, X otherwise" might get a lot of identical wrong submissions for X=1. It's useful to be able to find these.

I'm happy to write something up for this if it's useful.

— Reply to this email directly or view it on GitHub https://github.com/DOMjudge/domjudge/issues/185.

hex539 commented 8 years ago

The use-case we were thinking of was external feeds for people to make observations from. A new kind of wrong answer is something a commentator might want to commentate about.

That part is an API change only, which is perhaps worse -- I'm not sure how much flexibility there is for extra fields.

On DOMjudge I'd like to have the following:

Agree automatically grouping by default or showing obtrusive UI like bold fonts for new outputs wouldn't work very well, it hides too much useful information.

eldering commented 8 years ago

On the part of the API change: we can add things, but personally I'd rather not do so before carefully thinking about the need and the best way to implement this: if we do it, it should be right from the beginning. Also, we're trying to align our API to other systems, mainly Kattis, implementing ICPC specs (which are works in progress). That is not to say that we cannot add things on top.

For the jury interface, I think one simple approach would be to show on the submission page if and how many other submissions have an identical answer, and then add a link to a submission list (like submissions.php) filtered on that output. That sounds like a reasonable feature for the master branch to me.

Adding a "unique output" filter would then be fairly easy, although I think a bit of a special case. Not sure this is fit for master, but better for ICPC-live which we typically use for analyst work. It already has a few such extra features.

ubergeek42 commented 5 years ago

With the recent changes in domjudge and the improved API we have, this should be easier to implement as an external program reading off the api, and avoids the extra complexity this would add to the main codebase. There may be some missing API endpoints, but in that case we should improve the api(i.e. open a issue for any missing endpoints) rather than add this feature into the core of domjudge.