search

sourcegraph / cody

AI that knows your entire codebase
https://cody.dev
Apache License 2.0
2.22k stars 213 forks source link

feat(bench): unit test framework for cody-bench #4710

Open jamesmcnamara opened 3 days ago

jamesmcnamara commented 3 days ago

Adds a framework for evaluating "Generate Unit Test" performance through cody-bench. Reports a number of binary metrics like if expected test frameworks were used or if the test file was generated in the right place.

See https://github.com/sourcegraph/cody-leaderboard/pull/11 for a usage example

Test plan

Has been executed through https://github.com/sourcegraph/cody-leaderboard/pull/11

jtibshirani commented 2 days ago

Thanks @jamesmcnamara this is looking great! Did you see my comments above? After those are resolved (and other errors), looks ready to merge.