How to use this in continue with Ollama?

[Shadowfita]

I have the same question! Auto complete doesn't seem to be working for some reason.

[SlavikCA]

Here is my config:

  "models": [
    {
      "title": "qwen2.5-coder",
      "provider": "ollama",
      "model": "qwen2.5-coder:7b-instruct-q6_K",
      "apiBase": "http://192.168.0.44:11434/"
    }
  ],

Of course, the model has to be downloaded first.

Auto-complete doesn't work for me, too.

[huybery]

Please give us some time to give you all a best practice.

[dayekuaipao]

Here is my config:

  "models": [

    {

      "title": "qwen2.5-coder",

      "provider": "ollama",

      "model": "qwen2.5-coder:7b-instruct-q6_K",

      "apiBase": "http://192.168.0.44:11434/"

    }

  ],

Of course, the model has to be downloaded first.

Auto-complete doesn't work for me, too.

you should use base model for autocompleting, but it doesn't work too.

[SlavikCA]

So, the the rule is to use base model for autocompleting and instruct model for chat? Or can base model be used for chat, too?

[dayekuaipao]

So, the the rule is to use base model for autocompleting and instruct model for chat?

Or can base model be used for chat, too?

you should use instruct model for chat, it works. And you should use base model for autocompleting,it works for other model such as starcoder, but not for this model

[Shadowfita]

It's interesting because the readme talks about autocomplete support. I wonder if this is an issue with the continue.dev extension and the way it implements it, rather than the model.

[asmeurer]

FIM does seem to work with qwen2.5-coder-base:

$ollama run qwen2.5-coder:7b-base
>>> <|fim_prefix|>def fib(n):<|fim_suffix|>    return b<|fim_middle|>

   a = 0
   b = 1

   for i in range(2,n+1):
      c=a+b
      a=b
      b=c
<|endoftext|>class Solution(object):

(I control-C'd there because the ollama model doesn't seem to be configured to stop on <|endoftext|>).

I believe this needs to be added to the continue templates file https://github.com/continuedev/continue/blob/main/core/autocomplete/templates.ts

However, there may be a way to configure a template directly with continue https://docs.continue.dev/customize/config (maybe the tabAutocompleteOptions/template key?)

[rjmacarthy]

Hey all, FYI I've added support for this model using Ollama recently in https://github.com/twinnydotdev/twinny. Hope it helps someone. All the best.

[SlavikCA]

Looks like autocomplete for Qwen2.5-coder will be added to continue.dev, too:

https://github.com/continuedev/continue/issues/2330

[LNTH]

This is a workaround solution for autocomplete with vllm (I don't use ollama). I also found that Instruct model can work with autocomplete so I just have to load 1 model for both tasks (chat & autocomplete).

vllm serve Orion-zhen/Qwen2.5-Coder-7B-Instruct-AWQ --quantization awq --disable-log-requests --disable-log-stats

Continue config.json

{
  "models": [
    {
      "title": "Qwen2.5-Coder-7B-Instruct",
      "provider": "openai",
      "apiBase": "http://192.168.1.19:8000/v1/",
      "model": "Orion-zhen/Qwen2.5-Coder-7B-Instruct-AWQ"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen2.5-Coder-7B-Instruct",
    "provider": "openai",
    "apiKey": "None",
    "completionOptions": {
      "stop": [
        "<|endoftext|>",
        "\n"
      ]
    },
    "apiBase": "http://192.168.1.19:8000/v1/",
    "model": "Orion-zhen/Qwen2.5-Coder-7B-Instruct-AWQ"
  },
  "tabAutocompleteOptions": {
    "multilineCompletions": "never",
    "template": "You are a helpful assistant.<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
  },
  "customCommands": [],
  "allowAnonymousTelemetry": false,
  "docs": []
}

continuedev/continue#2388

Edit: correct config.json

[R-Dson]

This seems to work for me. The base model seems to work better from my testing.

  "tabAutocompleteModel": {
    "title": "qwen2.5-coder",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base-q4_K_M",
    "completionOptions": {
      "stop": [
        "<|endoftext|>"
      ]
    }
  },
  "tabAutocompleteOptions": {
    "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
  },

[dayekuaipao]

This seems to work for me. The base model seems to work better from my testing.

  "tabAutocompleteModel": {
    "title": "qwen2.5-coder",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base-q4_K_M",
    "completionOptions": {
      "stop": [
        "<|endoftext|>"
      ]
    }
  },
  "tabAutocompleteOptions": {
    "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
  },

cool, but how do you get the tabautocompleteoptions?

[R-Dson]

This seems to work for me. The base model seems to work better from my testing.

  "tabAutocompleteModel": {
    "title": "qwen2.5-coder",
    "provider": "ollama",
    "model": "qwen2.5-coder:7b-base-q4_K_M",
    "completionOptions": {
      "stop": [
        "<|endoftext|>"
      ]
    }
  },
  "tabAutocompleteOptions": {
    "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
  },

cool, but how do you get the tabautocompleteoptions?

I'm not sure what you are referring to, 'tabautocompleteoptions' is available in continue, if not then check if you need to update the extension. see https://docs.continue.dev/customize/deep-dives/autocomplete#tabautocompleteoptions for more info on 'tabautocompleteoptions'.

[caoxuhao]

I have the same question with ollama. You can change the modelfile and add stop paramter.

1. show the offical modelfile ollama show qwen2.5-coder:7b-base --modelfile
2. copy the modelfile to your local file
3. modifty the modelfile change the FROM sha-.... to FROM qwen2.5-coder:7b-base add new paramter PARAMETER stop <|endoftext|> after TEMPLATE {{ .Prompt }}

   FROM qwen2.5-coder:7b-base
   TEMPLATE {{ .Prompt }}
   PARAMETER stop <|endoftext|>
   ....

4. create new model image with your local file ollama create qwen2.5-coder:7b-base-fix -f your local file name
5. run the new model image ollama run qwen2.5-coder:7b-base-fix

[asmeurer]

People have also reported that upstream https://github.com/ollama/ollama/issues/7166. Hopefully it will be fixed (if they do, you can just ollama pull qwen2.5-coder:7b-base and it will download the updated modelfile).

[Umleitungen]

anyone know why I always get the following issues:

1. chat message prompted instead of code block in VsCode+Continue

2. sometimes, I get code without any new line break

"tabAutocompleteModel": {
    "model": "sparrowx-Qwen2.5-Coder-7B",
    "title": "sparrowx-Qwen2.5-Coder-7B",
    "provider": "openai",
    "apiBase": "https://xxxx",
    "completionOptions": {
      "stop": [
        "<|endoftext|>"
      ],
      "maxTokens": 5000,
      "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
     },
    "useLegacyCompletionsEndpoint": false
  }

[R-Dson]

anyone know why I always get chat message prompted instead of code block in VsCode+Continue

sometimes, I get code without any new line break

"tabAutocompleteModel": {
    "model": "sparrowx-Qwen2.5-Coder-7B",
    "title": "sparrowx-Qwen2.5-Coder-7B",
    "provider": "openai",
    "apiBase": "https://xxxx",
    "completionOptions": {
      "stop": [
        "<|endoftext|>"
      ],
      "maxTokens": 5000,
      "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
     },
    "useLegacyCompletionsEndpoint": false
  }

I assume that you want autocompletion across multiple lines, but continue do inline code suggestions by default I believe.

You can enable it to always do multiline completion by adding this line to the options

"tabAutocompleteOptions": {
   "multilineCompletions": "always"
 }

(From https://docs.continue.dev/autocomplete/how-to-customize)

[Umleitungen]

anyone know why I always get chat message prompted instead of code block in VsCode+Continue

sometimes, I get code without any new line break

"tabAutocompleteModel": {
    "model": "sparrowx-Qwen2.5-Coder-7B",
    "title": "sparrowx-Qwen2.5-Coder-7B",
    "provider": "openai",
    "apiBase": "https://xxxx",
    "completionOptions": {
      "stop": [
        "<|endoftext|>"
      ],
      "maxTokens": 5000,
      "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>"
     },
    "useLegacyCompletionsEndpoint": false
  }

I assume that you want autocompletion across multiple lines, but continue do inline code suggestions by default I believe.

You can enable it to always do multiline completion by adding this line to the options

"tabAutocompleteOptions": {
  "multilineCompletions": "always"
}

(From https://docs.continue.dev/autocomplete/how-to-customize)

it does not help on this case even I set multilineCompletions to always