Chat: add infrastructure to run chat evals

sourcegraph / cody

AI that knows your entire codebase

https://cody.dev

Apache License 2.0

2.24k stars 213 forks source link

Closed olafurpg closed 1 week ago

olafurpg commented 1 week ago

Fixes CODY-2548

Run pnpm agent:skip-root-build cody-bench --evaluation-config ~/dev/sourcegraph/cody-leaderboard/chat-bench.json
Confirm it generates this output here https://github.com/sourcegraph/cody-leaderboard/pull/3