Investigate using a customer's own LLM (chatGPT) contract

[malomarrec]

Problem

We've had requests from several customers and prospects about using their own openAI or Anthropic contract for Cody. We've reviewed and cleared from a legal standpoint, provided we add a clause in the order form.

Most recently, we heard this from https://github.com/sourcegraph/accounts/issues/2525. They have an existing contract with OpenAI and want to plug:

• embeddings
• chatGPT

into Cody.

We need to investigate how to do this. This has multiple dimensions:

• How can we plug chatGPT?
• Can we make the site-admin interface generic enough that a customer can point sourcegraph to their own instance themselves?
• Can we expose prompt engineering to the site-admin?
  1. we'll need it to make chatGPT vs Anthropic optimisations
  2. customers will need it eg. to customise it and/or prompt for their own way of doing things (eg. top 10 best practices)
  3. we'll need it for bring your own model

TODO

• [ ] spike / write docs
• [ ] legal review

Customers

• https://github.com/sourcegraph/accounts/issues/8004
• https://github.com/sourcegraph/accounts/issues/2525
• https://github.com/sourcegraph/accounts/issues/8660

Important talking point for:

• https://github.com/sourcegraph/accounts/issues/8661
• https://github.com/sourcegraph/accounts/issues/7485

[sqs]

Partial answer: Site config already supports using OpenAI for completions. The frontend needs to send slightly different params, such as the below:

const OPENAI_DEFAULT_CHAT_COMPLETION_PARAMETERS: Omit<CompletionParameters, 'messages'> = {
    temperature: 0.2,
    maxTokensToSample: SOLUTION_TOKEN_LENGTH,
}

(That change needs to be made in code currently.)

[vdavid]

The three parts of this issue, and their current status, with my thoughts:

1. Make the site admin interface generic enough that customers can point Sourcegraph to their own instance?

→ The site config keys completions and embeddings (documented here) already allow customers to use their own instance. Not sure if we need to do anything else.

2. How can we plug chatGPT?

→ We already support openai as a provider, albeit undocumented. The code might need some cleaning.

3. Can we expose prompt engineering to the site-admin?

Why?

• to optimize for ChatGPT vs. Anthropic
• to customize it for customers' own way of doing things (for example, top ten best practices)
• for "bring your own model"

→ a few thoughts:

• Prompt engineering is currently done on the front end, and the efforts to move it to the back end are paused. Because of this architecture, it'd be very ugly at this point to make it customizable. -- We probably want to fork our prompt engineering to maximize our prompts' efficiency with different LLMs. But I see little value in doing this at this stage. By using the same prompts for different LLMs, we're probably being 80% effective. Fine-tuning would require manual effort with our current tooling and could easily lead to regressions. I'd first invest in the tooling by setting up an automated prompt comparison environment to compare different prompts and models.

Next steps:

• Look into the openai client code
• Investigate potential tooling

[malomarrec]

Related: https://github.com/sourcegraph/cody/issues/221

[vdavid]

Update:

• Looked into the openai client code, but it's not in a terrible state and it works for now, so it didn't seem worthwhile to improve that
• I focused on a prompt tester tool that I've described here and that I'll demo an MVP for today at Demo day.

[mrnugget]

@vdavid

• [x] confirm whether it's backported
• [x] add docs

[swelborn]

Hello, cool work you all are doing here. Will this feature be supported for non-enterprise users? I would like to link up cody vscode extension to gpt-4.

[mrnugget]

Hello, cool work you all are doing here. Will this feature be supported for non-enterprise users? I would like to link up cody vscode extension to gpt-4.

Not sure yet! We're working on it and thinking about it. Supporting multiple LLMs is tricky, because you want to tweak prompts for each one. But get that folks want to use GPT4 (and others). We're trying to find a good solution here :)