[CON-227] Model Roles

[sestinj]

Validations

• [X] I believe this is a way to improve. I'll try to join the Continue Discord for questions
• [X] I'm not able to find an open issue that requests the same enhancement

Problem

There's currently no way to specify a model to use for Cmd+I or other specific features (currently it uses whatever is manually selected in the dropdown).

Solution

Add a "modelRoles" section to config.json that allows specification of model titles for each of several tasks. For example, this might look like:

{
  "models": [
    {
      "title": "Claude 3 Sonnet",
      "provider": "anthropic",
      "model": "claude-3-sonnet-20240229",
      "apiKey": "ANTHROPIC_API_KEY"
    },
    {
      "title": "Llama 3",
      "provider": "ollama",
      "model": "llama3"
    },
    {
      "title": "Starcoder 2 3b",
      "provider": "ollama",
      "model": "starcoder2:3b"
    }
  ],
  "modelRoles": {
    "cmdI": "Llama 3",
    "chat": "Claude 3 Sonnet",
    "autocomplete": "Starcoder 2 3b"
  }
}

_CON-227

[fbricon]

modelRoles in config.json should be the default for all IDEs, but the model roles should be overrideable in vscode settings.json and in jetbrains preferences respectively.

Some users prefer one IDE over the other, depending on the language they're developing in. e.g. Jetbrains for Java, VS Code for the rest. Some models might be specialized for specific language, so you'd want to use different ones, depending on context (thus IDE)

[fbricon]

I'd avoid using cmdI as it's a Mac-specific shortcut. interactive maybe?

[fbricon]

Model title as modelrole key is a no-go. When using ollama autodetect, the models are not listed in the config.json file. e.g. I have:

    {
      "model": "AUTODETECT",
      "title": "Ollama",
      "completionOptions": {},
      "apiBase": "http://localhost:11434",
      "provider": "ollama"
    }

But I can select:

You might want to support at least <model title>:<actual model key>

[buck-0x]

Model title as modelrole key is a no-go. When using ollama autodetect, the models are not listed in the config.json file. e.g. I have:

    {
      "model": "AUTODETECT",
      "title": "Ollama",
      "completionOptions": {},
      "apiBase": "http://localhost:11434",
      "provider": "ollama"
    }

But I can select:

You might want to support at least <model title>:<actual model key>

Agreed this is a problem. AUTODETECT is the way to go but it blocks using modelRoles.

[buck-0x]

Ideally we would be able to set modelRoles in the UI as well.

[sestinj]

@fbricon @buck-0x I just made this commit (https://github.com/continuedev/continue/commit/721ce6c7992de326f2a4e6bfb9c092247244ce6d?diff=unified&w=1) which introduces a model role called "inlineEdit", rather than "cmdI" for the reason you mentioned.

You might want to support at least :

If you are using AUTODETECT, you can follow the pattern used in the UI. For example, with "title": "Ollama" and the name of the autodetected model being "llama3:latest", you could set "inlineEdit": "Ollama - llama3:latest"

[fbricon]

I still think these properties should be overrideable per IDE. At the very least, provide the model role properties under an ide parent node, using keys as vscode, idea, ....

[fbricon]

Tried the pre-release. I selected an invalid model, then requested something, the extension didn't error, provided some response, but I don't know which model was used. So did it really work? was it accidental fallback? I think there should be some feedback, in the form of a notification, in case the model is invalid/unreachable