bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
830 stars 219 forks source link

Speedup execute.py: Reuse same manager and dict in #277

Open michaelfeil opened 1 month ago

michaelfeil commented 1 month ago

In my case, brought down the end-to-end latency from 3-5 seconds per execution to ~0.9 seconds.

Potentially needs a good review, I pulled this from an internal fork.