search

symflower / eval-dev-quality

DevQualityEval: An evaluation benchmark 📈 and framework to compare and evolve the quality of code generation of LLMs.
https://symflower.com/en/company/blog/2024/dev-quality-eval-v0.4.0-is-llama-3-better-than-gpt-4-for-generating-tests/
MIT License
137 stars 5 forks source link

fix, Score with passing tests in code-repair task cause coverage can be cheated #321

Closed bauersimon closed 3 months ago

bauersimon commented 3 months ago

TODO release notes of 0.6.1 and release once it is in

- fix, Score with passing tests in code-repair task cause coverage can be cheated by @bauersimon https://github.com/symflower/eval-dev-quality/issues/320, https://github.com/symflower/eval-dev-quality/pull/321

AND checkmark in roadmap https://github.com/symflower/eval-dev-quality/issues/195