Embedding provider other than transformer.js creates invalid index

Pawe98 commented 6 months ago

Before submitting your bug report

[x] I believe this is a bug. I'll try to join the Continue Discord for questions
[x] I'm not able to find an open issue that reports the same bug
[x] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS: Windows 11 Pro (Build 22631.3155, Windows Feature Experience Pack 1000.22684.1000.0)
- Continue: v0.8.2 - 2024-01-23 (Current stable version from VS Extensions)
- IDE: VS Code 1.87.0
- Model: Ollama any

Description

When im using a different embeddingsProvider other than the standard transformer.js, I get various bugs and an unexplained behavior. This is my setup:

Starting VS Code as a fresh install, adding the continue extension. Navigating into the config.js in users/USER/.continue/config.js and adding a custom embedding provider:

"embeddingsProvider": {
    "model": "codellama:7b",
    "apiBase": "http://localhost:11434",
    "provider": "ollama"
 }

Adding another Ollama "model" to be used for chat:

 "models": [
    {
      "model": "deepseek-coder:33b-instruct-q4_K_M",
      "title": "Ollama",
      "apiBase": "http://localhost:11434",
      "provider": "ollama"
    }
  ],

Closing VS Code, deleting the .continue/index directory. Downloading the necessary models with

  ollama pull codellama:7b
  ollama pull deepseek-coder:33b-instruct-q4_K_M

Starting ollama in a seperate cmd window with "ollama serve".

Now when im starting VS-Code, i can see that the index is created even before I see any logs of my ollama. This would tell me, that it is somehow creating the embeddings without using the specified embeddings provider. However, after a while I see that ollama is deploying the codellama model, and answer some api/embedding requests. The first few seem to be sucessfull:

However, after the first few it gives me errors:

(It looks like that continue makes requests for the embeddings, but just uses "something" else to create a broken index. Nonetheless, if i now write a message into the chat window, after all the requests are finished, ollama switches to the deepseek-coder model to generate a text response. However, if im now including the index via @CodeBase inside my message, I get an Error:

Error getting context items from codebase: TypeError: failed to downcast any to Array

this is the vs code dev tools, for some reason can't copy the console content:

To reproduce

Install all dependencies (ollama, required models)
Have ollama running in the background with ollama serve
Start VS Code with the continue extensions, adapt config file to use ollama as embeddingsProvider.
Restart VS Code (delete index manually, reload window, ...) --> A "broken" index is created, that probably doesen't even use the specified embeddingProvider, ollama embedding requests are sent but response timeout, index is created even before ollama embedding api is called.

Log output

The extension host log of vs code:

2024-02-29 20:07:43.038 [info] Extension host with pid 22320 started
2024-02-29 20:07:43.039 [info] Skipping acquiring lock for c:\Users\Paul\AppData\Roaming\Code\User\workspaceStorage\cd89fda7b8384bf3ea99dc178bb04111.
2024-02-29 20:07:43.110 [info] ExtensionService#_doActivateExtension Continue.continue, startup: false, activationEvent: 'onView:continue.continueGUIView'
2024-02-29 20:07:43.185 [info] ExtensionService#_doActivateExtension vscode.emmet, startup: false, activationEvent: 'onLanguage'
2024-02-29 20:07:43.355 [info] ExtensionService#_doActivateExtension vscode.git-base, startup: true, activationEvent: '*', root cause: vscode.git
2024-02-29 20:07:43.360 [info] ExtensionService#_doActivateExtension vscode.git, startup: true, activationEvent: '*'
2024-02-29 20:07:43.409 [info] ExtensionService#_doActivateExtension vscode.github, startup: true, activationEvent: '*'
2024-02-29 20:07:43.471 [info] Eager extensions activated
2024-02-29 20:07:43.507 [info] ExtensionService#_doActivateExtension vscode.debug-auto-launch, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:43.511 [info] ExtensionService#_doActivateExtension vscode.merge-conflict, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:43.517 [info] ExtensionService#_doActivateExtension ms-vscode-remote.remote-wsl-recommender, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:43.526 [info] ExtensionService#_doActivateExtension eamodio.gitlens, startup: false, activationEvent: 'onStartupFinished'
2024-02-29 20:07:56.468 [error] Error: `neon::types::Deferred` was dropped without being settled

Pawe98 commented 6 months ago

multiple models have been tested, this is the current config.json I use which does create the same behaviour as before:

{
  "models": [
    {
      "model": "codellama:7b",
      "title": "Ollama",
      "provider": "ollama"
    }
  ],
  "slashCommands": [
    {
      "name": "edit",
      "description": "Edit selected code"
    },
    {
      "name": "comment",
      "description": "Write comments for the selected code"
    },
    {
      "name": "share",
      "description": "Download and share this session"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    }
  ],
  "customCommands": [
    {
      "name": "test",
      "prompt": "Write a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "contextProviders": [
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "open",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "embeddingsProvider": {
    "model": "codellama:7b",
    "provider": "ollama"
  }
}

abbott commented 4 months ago

multiple models have been tested, this is the current config.json I use which does create the same behaviour as before:

Be sure to restart the extension once the new provider is added — Command Palette > Restart Extension Host

Here's a functioning embeddings provider using Voyage:

  "embeddingsProviders": {
    "provider": "openai",
    "model": "voyage-code-2",
    "apiBase": "https://api.voyageai.com/v1/",
    "apiKey": "REMOVED"
  },

catarino commented 3 months ago

I'm not getting embeddings to work.

I'm using LM Studio. Chat and tabAutocomplete are working as expected.

config.json file has:

{
  "title": "LM studio",
  "provider": "lmstudio",
  "model": "AUTODETECT",
  "apiBase": "http://localhost:1234/v1/",
  "apiKey": "lm-studio",
  "completionOptions": {}
}
…
 "tabAutocompleteModel": {
    "provider": "lmstudio",
    "title": "Codestral autocomplete",
    "model": "lmstudio-community/Codestral-22B-v0.1-Q8_0.gguf"
  },
  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-ai/nomic-embed-text-v1.5.Q8_0.gguf",
    "apiBase": "http://localhost:1234/v1"
  }

Albeit very slow, auto complete is working ok. but embeddings is showing this error:

models list is showing:

I also tried adding "apiKey": ""(with the actual key) changing the url to end with /v1/embeddings but also didn't work.

Any hints on what's going on?

catarino commented 3 months ago

Actually I think I found the problem:

Continue is looking for Ollama's http://127.0.0.1:1234/api/embeddings instead of http://127.0.0.1:1234/v1/embeddings used by LM Studio

I think this is another bug, probably should add it as issue. not sure @TyDunn ?

Henri-J-Norden commented 3 months ago

LM Studio uses OpenAI API format, try

"embeddingsProvider": {
  "provider": "openai",
  "model": "nomic-ai/nomic-embed-text-v1.5.Q8_0.gguf",
  "apiBase": "http://localhost:1234/v1/"
},

catarino commented 3 months ago

"model": "nomic-ai/nomic-embed-text-v1.5.Q8_0.gguf", "apiBase": "http://localhost:1234/v1/"

It works :)

the only thing I changed was "provider": "openai". I had "provider": "ollama" didn't know the provider part was so important.

Thanks a lot @Henri-J-Norden

sestinj commented 3 months ago

Thanks for helping out @Henri-J-Norden! Looks like both this and the original problem are resolved, so I'll close the issue

continuedev / continue