search

eecs-autograder / autograder.io

Central documentation and issue tracker for autograder.io
61 stars 9 forks source link

Stale stdout and stderr displayed after rerunning submission #25

Open CarsonHoffman opened 3 years ago

CarsonHoffman commented 3 years ago

Hi!

In the past few months of working with the Autograder and rerunning my own submissions (many times) while testing, I had noticed a phenomenon wherein certain test cases seemed to have become "stuck", specifically relating to their stdout and/or stderr output. I hadn't thought much of it, but this just happened to us on a student submission, so I thought it'd be good to file here.

This is specifically concerning project 1294, group 272723, submission 1229047, and test case 17325. Re-runs on this submission change the stdout and return code of this specific test case, but the stderr never changes across runs. (We're generating a long, unique string on each test case run, as well as printing out the current time, and nothing changes.) We're calling out to our own networked service which does the actual grading for this part, so we can confirm that the test case is actually running from an external perspective, and that the return code and stdout that we send back from our service is actually reflected in the Autograder (this submission is non-deterministic, which led us to discovering this), but the stderr simply never changes—this makes it confusing for staff on our end, as the stderr doesn't correspond to the score given to the student. This doesn't happen on other test cases on the same submission, even ones within the same suite.

I hope that this description was useful enough to describe the issue—please let me know if you have any questions or would like us to experiment with anything on our end. Thanks!

james-perretta commented 2 years ago

Hi Carson, That does sound very strange... Just to make sure I understand correctly, this only occurs on reruns, correct? And it's happened on other projects or just this one?

I think the first thing I would try is cloning the project to a local copy of the autograder and seeing if I can reproduce it there. This is unfortunately one of those "I have no idea why this would happen" things that will probably take a lot of guessing before stumbling into the actual cause. How urgent is this/do you have a way to work around it for the few students affected by this?

CarsonHoffman commented 2 years ago

As far as I can tell, this only happens on reruns, yes (as the symptom is not changing between reruns; we haven't seen the stdout/stderr be unexpectedly empty on a submission, for example). This isn't super urgent on our end as we don't expose stderr to students, only return code and stdout, but it may make for a confusing staff experience if the student happens to go to someone who isn't familiar with the situation and the grade seems inconsistent with the staff-visible output. I hadn't considered looking into this on a local copy, but I may try to do so if I can get things set up at some point 🙂

james-perretta commented 2 years ago

@CarsonHoffman As best I can tell, I think this might actually be an NFS cache staleness issue. I've observed what you describe recently, and then when I went back to check on it later the output reflected what I'd expect to see. I don't know enough about the details of NFS to know how to fix this, but perhaps someone else does?