Improvements for test cases

FherStk commented 9 months ago

When entering on 'edit test cases', if no data has been entered through the site but the test-cases have been created manually into the file system, a warning will be displayed:

Choosing the 'infer' option, the zip file will be used to create the test-cases (one entry per file, zero points per entry, default values).
Choosing the 'rebuilld' option, the yml file will be used to create the test-cases, but only works with simple problems.

Also, a new option has been added to the form:

If the zip file is requested to be deleted, all the test-cases will be also removed from the database.
If a zip file is uploaded, existing test-cases will be reused if possible (points, pretest and other configurations will be preserved), outdated cases will be removed, and the new ones will be added.

Another addition is the ability to automatically display the test-cases after the problem statement if needed:

All the test-cases will append after the problem statement automatically, avoiding inconsistencies between the files and the displayed data:

Notice that "pretests" will be displayed first (in order to avoid inconsistencies between the problem statement and the results sent to the judge) and, also, a new option has been added in order to make a test "private": imagen

Notice that private test will be displayed as hidden to avoid missing the judge test's correlation: imagen

All these changes are an addition, completely optional, and does not alter the default behavior of the base DMOJ.

I hope you like it, have nice holidays! Fher

codecov-commenter commented 9 months ago

Codecov Report

Attention: Patch coverage is 11.51832% with 169 lines in your changes missing coverage. Please review.

Project coverage is 46.32%. Comparing base (fd7fb05) to head (8fec955). Report is 15 commits behind head on master.

Files	Patch %	Lines
judge/models/problem_data.py	8.08%	125 Missing :warning:
judge/views/problem_data.py	9.37%	29 Missing :warning:
judge/views/problem.py	0.00%	6 Missing :warning:
judge/models/problem.py	37.50%	5 Missing :warning:
judge/utils/problem_data.py	0.00%	4 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #2305 +/- ## ========================================== - Coverage 46.76% 46.32% -0.45% ========================================== Files 251 253 +2 Lines 13317 13534 +217 ========================================== + Hits 6228 6269 +41 - Misses 7089 7265 +176 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

FherStk commented 8 months ago

Sorry, but I've found a little bug:

The "pretest" are executed prior to regular tests, so the input / output numeration between the statement and the judge results won't match.

I think that the best option to fix this is to implement this issue: https://github.com/DMOJ/online-judge/issues/2031

If you agree, I will create a new PR to implement https://github.com/DMOJ/online-judge/issues/2031 and, once integrated, this one will be ready to be integrated too.

Thanks and kind regards, Fher

FherStk commented 7 months ago

Sorry, but I've found a little bug:

When a test is marked as "pretest" it won't be displayed at the problem statement (if the option is selected within the problem settings), but "pretest" are executed prior to regular tests, so the input / output numeration between the statement and the judge results won't match.

I think that the best option to fix this is to implement this issue: #2031

If you agree, I will create a new PR to implement #2031 and, once integrated, this one will be ready to be integrated too.

Thanks and kind regards, Fher

Well... I see that the #2031 issue is something bigger than I imagined, and I don't feel ready to address it right now. Maybe I should reimagine how the pretest behavior should be treated because:

Pretests are executed before the regular tests.
Pretests exists in order to execute them exclusively during a contest, so the regular tests won't be executed till the end of the contest. Does this mean that regular test's inputs and outputs should be hidden during a contest?
If the answer to the prior question is "yes", this means that only pretest's inputs and outputs should be displayed within the problem statement, but it has no sense to mark all tests as pretests in order to display them all.
So, arrived to this point, maybe should be better to add a new column to mark a test (or pretest) as hidden, but this will not solve the reordering problem when a pretest is selected... I wouldn't like the idea of disabling pretest selection when a problem enabled the "include test cases" option...
Finally, I think that the best option would be to list first the pretests and then the regular test, in the same order as they are but grouped. If a test must be hidden from the statement, it should be marked as it (new "hidden test" option) and a caption like "This is a hidden input / output" will be displayed in the problem statement to avoid gaps.

Sorry, I have so many doubts ^_^U

FherStk commented 1 month ago

Hi there!

Sorry, I've been a bit out but I'm now ready to finish this implementation. I decided to proceed adding a new column to tests in order to mark them as private, so those private tests's inputs and outputs wont be displayed at the problem statement. Also, pretests will be displayed prior to regular tests in order to avoid correlation issues between the judge results and the statement order.

I also made changes in my devel environment so it will be easier for me to squash the commits and send to you the PR with no conflicts.

Please, feel free to contact me if you need further information. Cheers! Fher

FherStk commented 1 month ago

Hi again,

PR ready to review and merge (if you like). I also updated the first post, so all the changes are described there.

Cheers! Fher

DMOJ / online-judge

Improvements for test cases #2305

Codecov Report