bigcode-project / bigcode-evaluation-harness

A framework for the evaluation of autoregressive code generation language models.
Apache License 2.0
781 stars 208 forks source link

Speedup execute.py: Reuse same manager and dict in #277

Open michaelfeil opened 2 days ago

michaelfeil commented 2 days ago

In my case, brought down the end-to-end latency from 3-5 seconds per execution to ~0.9 seconds.

Potentially needs a good review, I pulled this from an internal fork.