symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
57 stars 3 forks source link

Fixed timeouts for `symflower unit-tests` and `symflower test` #167

Open Munsio opened 3 weeks ago

Munsio commented 3 weeks ago

During the run of the light repository with the ollama models I encountered a problem where symfower test was running for ~1H (stuck). In order to avoid this we need to set sane timeouts for all those execution calls.

Notes:

Munsio commented 3 weeks ago

@bauersimon @ahumenberger @ruiAzevedo19 I've added myself here as assignee but everyone can take this Issue. I also believe we should fix this sooner then later as it is currently blocking the execution of v0.5.0

Munsio commented 3 weeks ago

language/java/language.go:96 language/golang/language.go:100 model/symflower/symflower.go:42

I guess those are the parts where we need to apply the timeout.