Cody-bench: add chat context

sourcegraph / cody

Type less, code more: Cody is an AI code assistant that uses advanced search and codebase context to help you write and fix code.

https://cody.dev

Apache License 2.0

2.59k stars 275 forks source link

Cody-bench: add chat context #4748

Closed janhartman closed 2 months ago

janhartman commented 3 months ago

Results are shown in https://github.com/sourcegraph/cody-leaderboard/pull/14.

If context is defined in the benchmark options, we index context.sourcesDir and retrieve context items from it. The indexing needs to be performed only once so it's done on startup and skipped for subsequent runs.

Test plan

N/A

github-actions[bot] commented 3 months ago

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

janhartman commented 2 months ago

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

This is a false positive, I just modified a related line :)

jtibshirani commented 2 months ago

@janhartman a couple questions:

Are we only covering PLG context retrieval (symf plus embeddings) here?
How are you thinking of keeping this in sync with https://github.com/sourcegraph/context-eval, which is the "ground truth" for chat context eval?

janhartman commented 2 months ago

@jtibshirani I tested PLG retrievers here as this is my primary focus right now, but we should totally add enterprise context instead. We have some plans at the hackathon this week that might enable enterprise context for OSS repos on dotcom as well :) As for your second point, no plans so far. I am however putting together a new eval framework that should be a merge of context-eval and Hitesh's framework (i.e. it will support all our retrievers) and more metrics than just recall. Hopefully, with the move to server-side context, the new framework will be all we need to do context evals and maybe the functionality here will be irrelevant.