search

pc2ccs / pc2v9

Version 9 of the PC^2 Programming Contest Control System
Eclipse Public License 2.0
46 stars 23 forks source link

Shadow should skip submissions after EOC #399

Open clevengr opened 2 years ago

clevengr commented 2 years ago

Describe the issue: Currently the Shadow RemoteEventFeedMonitor code submits to the PC2 Server every submission which it finds in the Primary (Remote) CCS Event Feed. Some CCS's (Kattis is one such example; PC2 is another) accept submissions after the end of the contest -- they just don't consider them in computing the standings. Kattis does however send these submissions in the event feed (PC2 may also do this; hasn't been verified). However, Kattis (at least; possibly PC2 as well) does not send any "judgement" element in the event feed for such runs.

This means that the PC2 Shadow will receive an EF event for a submission which came after the EOC. It will then forward this submission to the PC2 Server. The PC2 Server will mark it with state "DEL" (deleted) -- but it will still allow the Judges (AJ or human) to judge it. The effect of this is that PC2 will contain a run which has a judgement but will never get a corresponding judgement from the remote CCS. This in turn generates a guaranteed mismatch in the "Run Compare" display of the Shadow, and if PC2 judged the run as "Yes" then it will also generate a mismatch in the Shadow "Scoreboard Compare".

Submissions which are not considered part of the contest should not be considered in Shadow comparisons.

Additional context: Two possible solutions come to mind: 1 - in RemoteEventFeedMonitor, check the contest submission time each submission; if it is past the end of contest time, do not send the submission to the PC2 server (skip the call to submitter.submitRun). 2 - in ShadowCompareRunsPane, check each run (Submission) to see if it has been marked as Deleted ("DEL") by PC2; if so, don't display it. Note that this also requires considering what to do about the "run counts" at the bottom of that screen; it also requires considering what the effect of runs which exist in the PC2 server (and may have been judged, either Yes or No) is on the Scoreboard Compare screen.

lane55 commented 2 years ago

All submission from the Primary CCS should be submitted and save in the DB.

The processing of runs by the shadow compares/etc code should change so runs that are deleted, like everywhere else in the system, are not processed. That consistency means we do not have to remember two or more ways to handle deleted runs. Also, what happens if during the contest a run is removed by the Primary CCS? We can mark it deleted but if runs are skipped there is no code in the Shadow (as there should be) to ignore deleted runs.

That approach also is easier (less setup, more atomic) to unit test.

How can one answer the question how many runs were submitted after the end of the contest using the UI if they are never added in the system?

lane55 commented 1 year ago

The fix for solution number 2 is coded.

Pushed code that does not add deleted runs in the pc2 runs map on branch lane55 for branch i_399_ignore_deleted_or_EOC_runs_when_comparing_runs

in ShadowCompareRunsPane, check each run (Submission) to see if it has been marked as Deleted ("DEL") by PC2; if so, don't display it. Note that this also requires considering what to do about the "run counts" at the bottom of that screen; it also requires considering what the effect of runs which exist in the PC2 server (and may have been judged, either Yes or No) is on the Scoreboard Compare screen.

Did not change ShadowCompareRunsPane, changed the data that is fetched by ShadowCompareRunsPane.

As a result of the pane only getting non-deleted runs, the run counts should be correct, further there is no effect whatsoever on the deleted run's judgement (because they are not included)