search

Kattis / problemtools

Tools to manage problem packages using the Kattis problem package format.
MIT License
105 stars 72 forks source link

Safety margin can be shown wrong with fixed time limits #151

Open thorehusfeldt opened 4 years ago

thorehusfeldt commented 4 years ago

timelim_margin may be a float (because safety_margin can be set to, say, 1.2 in problem.yaml), it is not rounded to int in this case:

https://github.com/Kattis/problemtools/blob/063b68a98237773a11555c14fd2da07a25f68860/problemtools/verifyproblem.py#L1380

When the new time limit is announced to the user a few lines below, this is (implicitly) rounded downwards because the string expression'%d' % 2.4' apparently gives '2' (while'{:d}.format(2.4)' would have crashed).

Should just be timelim_margin = int(timelim * safety_margin) or maybe timelim_margin = round(timelim * safety_margin) or timelim_margin = int(.5 + timelim * safety_margin).

I’ll fix this (as part of a larger cleanup).

austrin commented 4 years ago

Sounds good, FTR rounding it to nearest integer sounds like the correct thing.

RagnarGrootKoerkamp commented 3 years ago

I think I've just been hit by this:

   Slowest AC runtime: 0.492, setting timelim to 1 secs, safety margin to 2 secs
...
WARNING in submissions: TLE submission mees-exclude-random-quadrangles.py (Python 3 w/Pypy) sensitive to time limit: limit of 1 secs -> TLE, limit of 2 secs -> AC
   TLE submission mees-exclude-random-quadrangles.py (Python 3 w/Pypy) OK: TLE [test case: test case secret/wide, CPU: 2.43s @ test case secret/random-max-7]

The safety margin is reported to be 2s, but a TLE submission with runtime of 2.43s still give a warning.

thorehusfeldt commented 3 years ago

By the way, no fix from my side is imminent, and the “larger cleanup” has stalled. I am still stumped by various definitions of “what a grade/judgement/verdict/score is” and can’t become constructive.