search

dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Analyse overhead for TESTed in Dodona #333

Closed pdawyndt closed 1 year ago

pdawyndt commented 1 year ago

TESTed-Docker seems to have a startup overhead of 10 seconds in Dodona. Take action to reduce this overhead.

niknetniko commented 1 year ago

Detailed timings locally revealed that creating a Docker containers is instant (as it is not started yet). Additionally, the runtime reported by Dodona includes starting the Docker container: for example, a runtime of 2 seconds includes starting the Docker container.

What did take some time though was the prepare (and finalise step), in which the judge repo is copied to the workdir. To check if this is due to the works, I executed the same exercise on Naos:

Naos Dodona
Kolomtranspositie (TESTed) 7s (1.9s runtime) 11s (1.4s runtime)
Zonnestelsel (JS) 7s (0.6s runtime) 4s (0.8s runtime)

There is a much bigger variance in Dodona runs however, sometimes taking 30s to see the submission while the runtime is 4s.

However, the difference between TESTed and JS on Dodona while being virtually the same Naos probably means the copying of the files on the workers takes a significant amount of time. Should this be the case, we can either try to reduce the size of the TESTed repository or cache the judge files on the workers locally.

bmesuere commented 1 year ago

Is tested that much bigger/more files than the other judges? I would expect that NFS from a neighbouring server would be fast enough not to have te count in seconds. (Especially because the exercise files are loaded from the same server)

niknetniko commented 1 year ago

Is tested that much bigger/more files than the other judges? I would expect that NFS from a neighbouring server would be fast enough not to have te count in seconds. (Especially because the exercise files are loaded from the same server)

There is: git ls-files | wc -l gives 1219 for TESTed and 17 for the JavaScript judge.

However, before changing stuff on the workers, we can look into optimising the repo a bit too see if it is enough. For example, the benchmarking folder contains 820 files, which could be moved to another repo relatively easy. There are also 246 files in the exercises folder, some of which are probably not necessary for the tests to run.

bmesuere commented 1 year ago

This would be great news since with some git trickery, it would solve the delay for all languages without functional impact.

niknetniko commented 1 year ago

Reducing the number of files to 342 doesn't seem to have done much, so I will continue to investigate.

niknetniko commented 1 year ago

Fixed for now by running git gc, but issue to track permanent fix: https://github.com/dodona-edu/dodona/issues/4578

niknetniko commented 1 year ago

This still seems to be an issue for some exercises, e.g. https://dodona.ugent.be/nl/activities/564920538/

niknetniko commented 1 year ago

Another culprit seems to be the handling of the partial output format. From some local timings (https://github.com/dodona-edu/dodona/issues/4579)

image

niknetniko commented 1 year ago

Beside a few small things for which issues exist here, most of the slowdowns are gone.