Display warning when grader compilation fails when testing submissions

[Binaryvector]

When running problemtools with the flag -p submissions, no warning or any other message is displayed, when the grader program fails to compile. Instead, problemtools silently switches to the default grader.

Currently an error is only displayed when also adding graders to the problem part flag, but I think it makes sense to add a warning even if that flag is not used, because: 1) When testing only submissions, the grader program has to be compiled anyway, so one might as well display a warning if the compilation fails. 2) Runtime errors in the grader cause a JE, i.e. they are considered errors when testing submissions. So maybe a compilation error should be considered an error too? 2) Since the verdict of a submission depends on the used grader, I think that information is important to be displayed when testing submissions. Currently problemtools just fails silently and uses the default grader instead, which might give the wrong verdict.

[ghamerly]

@Binaryvector can you provide a minimal problem package that demonstrates these issues (and how you test them)?

[Binaryvector]

https://www.dropbox.com/s/f4jmutiucduj4km/hello.zip?dl=0 This is based on your hello world example. I changed the problem specification to be a scoring problem and added a custom grader that has an error.

Without the error, the grader would give every submission a TLE. Problemtools does not use this grader because of the error, so it scores everything as AC instead.

verifyproblem hello -p submissions Here problemtools does not display anything regarding the grader and just silently uses the default grader.

Loading problem hello
Checking submissions
   AC submission hello.cc (C++) OK: AC (0) [CPU: 0.00s @ test case secret/hello]
   AC submission hello.py (Python 2) OK: AC (0) [CPU: 0.01s @ test case secret/hello]
   AC submission hello_alarm.c (C) OK: AC (0) [CPU: 1.00s @ test case secret/hello]
   Slowest AC runtime: 1.003, setting timelim to 5 secs, safety margin to 10 secs
hello tested: 0 errors, 0 warnings

verifyproblem hello -p submissions graders Now problemtools throws an error regarding the grader.

Loading problem hello
Checking submissions
   AC submission hello.cc (C++) OK: AC (0) [CPU: 0.00s @ test case secret/hello]
   AC submission hello.py (Python 2) OK: AC (0) [CPU: 0.01s @ test case secret/hello]
   AC submission hello_alarm.c (C) OK: AC (0) [CPU: 1.00s @ test case secret/hello]
   Slowest AC runtime: 0.997, setting timelim to 5 secs, safety margin to 10 secs
Checking graders
ERROR in graders: Compile error for grader.py (Python 2):
          File "/tmp/verify-hello-Y0KCs2/grader.py/grader.py", line 7
            print "TLE 0"
                ^
        SyntaxError: invalid syntax
hello tested: 1 error, 0 warnings

When I fix the grader by commenting the line 'someError(', then verifyproblem hello -p submissions returns the expected TLE verdicts

Loading problem hello
Checking submissions
ERROR in submissions: AC submission hello.cc (C++) got TLE [test case: test case secret/hello, CPU: 0.00s @ test case secret/hello]
ERROR in submissions: AC submission hello.py (Python 2) got TLE [test case: test case secret/hello, CPU: 0.01s @ test case secret/hello]
ERROR in submissions: AC submission hello_alarm.c (C) got TLE [test case: test case secret/hello, CPU: 1.01s @ test case secret/hello]
   Slowest AC runtime: 1.012, setting timelim to 5 secs, safety margin to 10 secs
hello tested: 3 errors, 0 warnings