search

jcrodriguez-dis / moodle-mod_vpl

Virtual Programming Lab for Moodle (Module)
GNU General Public License v3.0
103 stars 89 forks source link

'Evaluate All' sometimes skips submissions #192

Open FeldrinH opened 6 months ago

FeldrinH commented 6 months ago

In certain cases, running 'Evaluate' -> 'All' in the 'Submission list' tab appears to complete successfully, but looking at the list of submissions afterwards, it appears that some of them were skipped and not evaluated.

In particular, this seems to happen if submissions are sorted by grade and evaluating a submission changes the grade (for example, because the test cases have changed). I suspect what is happening is that 'Evaluate All' evaluates the submissions in the order that I have selected (sorted by grade) and it always uses the latest ordering, so as the grades change with evaluation the order also changes and so the system ends up skipping over some submissions and evaluating some submissions twice.

This is somewhat annoying because it is unexpected and makes 'Evaluate All' unreliable.

FeldrinH commented 6 months ago

PS: I don't currently have a good set of submissions to test this on, but IIRC I have reproduced this with VPL 4.2.3 as well as several older versions.

jcrodriguez-dis commented 6 months ago

Dear @FeldrinH,

Thank you for bringing this issue to our attention.

We recognize that this problem has persisted for a considerable time. We think that the current batch process is inefficient and requires a complete overhaul. We will prioritize this issue to give a solution.

Best regards, Juan Carlos

FeldrinH commented 6 months ago

We recognize that this problem has persisted for a considerable time. We think that the current batch process is inefficient and requires a complete overhaul. We will prioritize this issue to give a solution.

That would be great. If an overhaul is coming then it would be really nice if the batch processing could evaluate multiple submissions in parallel. Currently it seems to evaluate one at a time while most CPU cores in our jail server sit idle.