dodona-edu / universal-judge

Universal judge for educational software testing
https://docs.dodona.be/en/tested
MIT License
9 stars 5 forks source link

Don't use Mako templates to generate code #334

Closed niknetniko closed 1 year ago

niknetniko commented 1 year ago

I did some profiling of TESTed with the "Star Battle" exercise (in JavaScript). This showed that of the original ~5s execution time, only half was spent executing the actual submission, while half was spent in the Mako templates generating code.

In this PR, I removed the use of Mako to generate code, and used manual string construction instead. Hyperfine results of TESTed with Mako templates:

Benchmark 1: python -m tested.manual
  Time (mean ± σ):      5.561 s ±  0.327 s    [User: 5.417 s, System: 0.208 s]
  Range (min … max):    4.972 s …  6.306 s    10 runs

The same exercise without Mako templates:

Benchmark 1: python -m tested.manual
  Time (mean ± σ):      1.972 s ±  0.276 s    [User: 1.943 s, System: 0.084 s]
  Range (min … max):    1.709 s …  2.380 s    10 runs

So, to summarize:

Given that speed is important, I think the trade-off with slightly uglier code generation is worth it.

Fixes #332.

pdawyndt commented 1 year ago

Good job! Has this any impact on the programming language configurations themselves, i.e. do they need to be updated one by one?

niknetniko commented 1 year ago

Good job! Has this any impact on the programming language configurations themselves, i.e. do they need to be updated one by one?

Yes, as this change is purely in the programming languages modules (i.e. there are practically no changes to the core of TESTed). However, I already converted all modules in this PR.