Autocomplete on vscode stops working

longle255 commented 3 weeks ago

Before submitting your bug report

[X] I believe this is a bug. I'll try to join the Continue Discord for questions
[X] I'm not able to find an open issue that reports the same bug
[X] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:macOs
- Continue version: 0.9.215
- IDE version: 1.94.0
- Model: any (qwen2.5-coder:7b-base, deepseek-coder:6.7b-base)
- config.json:

{
  "models": [
    {
      "title": "Ollama",
      "provider": "ollama",
      "model": "AUTODETECT"
    },
    {
      "title": "qwen2.5-coder:7b-instruct",
      "model": "qwen2.5-coder:7b-instruct",
      "contextLength": 16384,
      "apiBase": "http://localhost:11434",
      "provider": "ollama"
    }
  ],
  "customCommands": [
    {
      "name": "test",
      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "tabAutocompleteModel": {
    "title": "qwen2.5-coder:7b-base",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base"
  },
  "tabAutocompleteOptions": {
    "disableInFiles": ["*.md", "*.txt"]
  },
  "completionOptions": {
    "maxTokens": 8192
  },

  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    }
  ],
  "slashCommands": [
    {
      "name": "edit",
      "description": "Edit selected code"
    },
    {
      "name": "comment",
      "description": "Write comments for the selected code"
    },
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ],
  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-embed-text"
  }
}

Description

Autocomplete worked for me before, until recently (yesterday or today, I'm not sure when), and then it stopped working. The chat feature is still working fine with any models. Only autocomplete has the problem.

Output data of autocomplete shows it did not generate the prompt. The output of ollama shows no prompt was provided.

I tried a couple of versions of Continue (release, pre-release), but the problem persists.

To reproduce

No response

Log output

Continue log

==========================================================================
==========================================================================
##### Completion options #####
{
  "contextLength": 8096,
  "maxTokens": 8192,
  "model": "qwen2.5-coder:7b-base",
  "temperature": 0.01,
  "stop": [
    "<|endoftext|>",
    "<|fim_prefix|>",
    "<|fim_middle|>",
    "<|fim_suffix|>",
    "<|fim_pad|>",
    "<|repo_name|>",
    "<|file_sep|>",
    "<|im_start|>",
    "<|im_end|>",
    "\n\n",
    "\r\n\r\n",
    "/src/",
    "#- coding: utf-8",
    "```",
    "\nfunction",
    "\nclass",
    "\nmodule",
    "\nexport",
    "\nimport"
  ],
  "raw": true
}

##### Prompt #####
==========================================================================
==========================================================================
Completion:

output from ollama showing no prompt entered

DEBUG [process_single_task] slot data | n_idle_slots=4 n_processing_slots=0 task_id=1 tid="0x1ebeefac0" timestamp=1728402861
[GIN] 2024/10/08 - 17:54:21 | 200 |   26.967541ms |       127.0.0.1 | POST     "/api/generate"
time=2024-10-08T17:54:21.898+02:00 level=DEBUG source=sched.go:407 msg="context for request finished"
time=2024-10-08T17:54:21.898+02:00 level=DEBUG source=sched.go:339 msg="runner with non-zero duration has gone idle, adding timer" modelPath=/Users/test/.ollama/models/blobs/sha256-485c567a3dcfc5403695bf0d70b3b70c0df36034426f046a35eabf9d24676878 duration=30m0s
time=2024-10-08T17:54:21.898+02:00 level=DEBUG source=sched.go:357 msg="after processing request finished event" modelPath=/Users/test/.ollama/models/blobs/sha256-485c567a3dcfc5403695bf0d70b3b70c0df36034426f046a35eabf9d24676878 refCount=0
time=2024-10-08T17:54:23.281+02:00 level=DEBUG source=sched.go:575 msg="evaluating already loaded" model=/Users/test/.ollama/models/blobs/sha256-485c567a3dcfc5403695bf0d70b3b70c0df36034426f046a35eabf9d24676878

Patrick-Erichsen commented 3 weeks ago

Hi @longle255 , any chance you are seeing the same error reported here? https://github.com/continuedev/continue/issues/2457#issuecomment-2400013843

longle255 commented 3 weeks ago

@Patrick-Erichsen I can't tell for sure if the causes of those errors was the same. I don't see similar error message in vscode debug console. in fact I don't see any error at all, in any output

alanxmay commented 3 weeks ago

@Patrick-Erichsen I'm facing same issue, the reason why the completion was a space character is because the stop list contains "```".

And Qwen2.5-Coder-7B-Instruct completion result is

 ```python\n    # Uncomment the following line to set a custom chat template\n     # runtime.endpoint...

Modify stop list and it works. But I didn't figure out how to config continue to make this works.

facelezzzz commented 3 weeks ago

Same with Intellij IDEA

##### Completion options #####
{
  "contextLength": 8192,
  "maxTokens": 8192,
  "temperature": 0,
  "topP": 1,
  "presencePenalty": 0,
  "frequencyPenalty": 0,
  "model": "deepseek-chat",
  "stop": [
    "<｜fim▁begin｜>",
    "<｜fim▁hole｜>",
    "<｜fim▁end｜>",
    "//",
    "<｜end▁of▁sentence｜>",
    "\n\n",
    "\r\n\r\n",
    "/src/",
    "#- coding: utf-8",
    "```",
    "\nclass",
    "\nfunction"
  ],
  "raw": true
}

##### Prompt #####

longle255 commented 3 weeks ago

After some debugging into continue code, I think I found the cause of my issue, it's the contextLength of tabAutocompleteModel.

Looks like by default the contextLength for tabAutocompleteModel is 8096, while my option for "completionOptions": { "maxTokens": 8192 }. That make the prompt to be pruned completely.

So my fix was adding contextLength option to tabAutocompleteModel

"tabAutocompleteModel": {
    "title": "qwen2.5-coder:7b-base",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base",
    "contextLength": 16384
  },
  "completionOptions": {
    "maxTokens": 8192
  }

this fixes the problem for me.

@facelezzzz I think it would also fix the problem for you, as your context length and maxToken were the same.

facelezzzz commented 3 weeks ago

@longle255 It works again! Thank you for your efforts.

Patrick-Erichsen commented 2 weeks ago

Thanks for the writeup here @longle255 ! We should definitely provide a better warning in this case.

Curious why both of you are explicitly setting your completionOptions to 8192?

longle255 commented 2 weeks ago

@Patrick-Erichsen As far as I remember, I had a problem with Continue where the the generated answer was cut off in the middle for quite a few times for me. After digging a while I found an issue here talking about the same problem https://github.com/continuedev/continue/issues/2319 and they mentioned about the option in one of the comment. Adding the option did fix the problem for me.

continuedev / continue