Codeforces / polygon-issue-tracking

polygon-issue-tracking
16 stars 2 forks source link

Automatically set solution tags based on invokation #525

Closed kunyavskiy closed 9 months ago

kunyavskiy commented 10 months ago

Solution tags is an important feature to check, that all solutions have expected verdicts. When you are developing the problem, you mostly understand what is expected from solution. On the other side when adding an archive problem, it's not always obvious. Also, setting tags for all subgroups can be annoying, if there are many groups and solutions.

So I'd suggest to have an automation, that sets solutions tags based on invocation. Although, it can be not as accurate as manual setting, it would allow to fix current state and get notified if something changes. Also, this would be a nice way to compactly see how solutions are failing on which group.

Formally, what do I suggest

  1. After invocation without FL verdicts happen, there is a button "Set tags based on this invocation" appears.
  2. For each solution, except main one, global tag is set to
    • (Optional, not sure if it's good idea) Timelimit or correct if only ok and TL exists, but there are >0.9 of limit oks or only <1.1 of limit TLS.
    • Correct tag, if solution passes all tests
    • Single-wrong-verdict tag (like Wrong answer/Time limit) if applicable exists
    • Several-wrong-verdict tag (like Time or Memory limit exceeded) if applicable exists
    • Incorrect tag, if none of previous is applicable
      1. If groups are enabled in the problem, the same happens for each group tag on each solution.
        • Here is a tricky case for group dependencies. Failed verdicts should be checked not only in group tests, but also in it's dependencies.
geranazavr555 commented 9 months ago

Implemented. To try it: