The qwen2.5-coder:7b-base FIM model is not working in combination.

[00010110]

Before submitting your bug report

• [ ] I believe this is a bug. I'll try to join the Continue Discord for questions
• [ ] I'm not able to find an open issue that reports the same bug
• [ ] I've seen the troubleshooting guide on the Continue Docs

Relevant environment info

- OS:linux
- Continue:v0.9.210 (pre-release)
- IDE:vscode 1.88.0
- Model:qwen2.5-coder:7b-base
- config.json:

Description

On the same configuration, with the latest Codestral version and FIM model, it can work normally.

To reproduce

No response

Log output

==========================================================================
==========================================================================
Settings:
contextLength: 8096
model: qwen2.5-coder:7b-base 
maxTokens: 2048
temperature: 0.01
stop: <fim_prefix>,<fim_suffix>,<fim_middle>,<file_sep>,<|endoftext|>,</fim_middle>,</code>,

,

,/src/,#- coding: utf-8,,
function,
class,
namespace,
use
raw: true
log: undefined

############################################

<fim_prefix>    {
        /**
         * 删除上线单
         */
        $postData = I("post.");
        $DeployLogic = new \Admin\Logic\DeployMoreLogic();
        $result = $DeployLogic->delTask($postData);
        Logger::info(S("master_releasing"));
        if ($result !== false) {
            $this->ajaxReturn($this->get_json(0, "删除成功", ""), 'eval');
        } else {
            Logger::err("delTask接口调用失败");
            $this->ajaxReturn($this->get_json(10001, "接口调用失败", ""), 'eval');
        }
    }
    public function release()
    {
        /**
         * 上线
         */
        $isStart = [];
        $postData = I("post.");
        $DeployLogic = new \Admin\Logic\DeployMoreLogic();
        $result = $DeployLogic->release($postData, $isStart);
        if ($result !== false) {
            $this->ajaxReturn($this->get_json(0, "上线任务开始成功", ""), 'eval');
        } else {
            Logger::err("release接口调用失败");
            $this->ajaxReturn($this->get_json(10001, $isStart['errMsg']?$isStart['errMsg']:"接口调用失败", ""), 'eval');
            if(isset($isStart['isLocked']) && $isStart['isLocked']){
                S("master_releasing", true);
                Logger::info(S("master_releasing"));
                if(<fim_suffix>)

        }
    }

    private function get_json($errCode = 0, $errMsg = "", $data)
    {
        /**
         * json encode
         */
        $result["ret"] = $errCode;
        $result["errmsg"] = $errMsg;
        $result["data"] = $data;<fim_middle>

[00010110]

my Autocomplete config "tabAutocompleteModel": { "title": "qwen2.5", "provider": "ollama", "apiBase": "http://192.168.13.90:11434", "model": "qwen2.5-coder:7b-base " }

[cbingrui]

Could be: "template": "<|fim_prefix|>{{{ prefix }}}<|fim_suffix|>{{{ suffix }}}<|fim_middle|>", Referenced from: https://github.com/QwenLM/Qwen2.5-Coder?tab=readme-ov-file#3-file-level-code-completion-fill-in-the-middle

[0h41]

https://arxiv.org/html/2409.12186v1 You can see there that FIM tokens indeed changed compared to codeqwen, so @cbingrui template is correct. Works for me also. But even more important: you should use -instruct, not -base model. I've not been sure either, but then checked it by manually prompting both models with FIM examples in ollama. Given <|fim_prefix|>Hello<|fim_suffix|>ld!<|fim_middle|>, base model just starts spewing a lot of code. The same prompt result in actual line from -instruct. It makes sense if you think about it, base model is just that - does not include training with FIM tokens.

[KangmoonSeo]

I had the same issue, but after following the qwen2.5-coder instructions and making some adjustments, it worked. @cbingrui I'll submit a PR with the fix soon.

[harrychin]

That fix isn't quite complete, Qwen 2.5 coder seems to have another stop token still defined: "<|endoftext|>"

This particularly seems to occur around attempts to create strings/comments.

[KangmoonSeo]

@harrychin Thank you for catching this! You're right, I missed the <|endoftext|> stop token at #2341 .

I've also confirmed this in the tokenizer_config.json file on the Hugging Face repo. (https://huggingface.co/Qwen/Qwen2.5-Coder-7B/blob/main/tokenizer_config.json)

I'll create a new PR to fix this shortly. Really appreciate your thorough review and feedback.

EDIT: Will create PR for branch cleanup after updating dev

[sestinj]

PR has been merged, thank you @KangmoonSeo !