search

twinnydotdev / twinny

The most no-nonsense, locally or API-hosted AI code completion plugin for Visual Studio Code - like GitHub Copilot but completely free and 100% private.
https://twinny.dev
MIT License
2.91k stars 153 forks source link

Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. #314

Open machisuji opened 2 weeks ago

machisuji commented 2 weeks ago

Describe the bug Try using twinny with deepseek-coder-v2:16b via ollama, but both chat and FIM don't seem to work due to the following errors seen in the ollama logs:

[GIN] 2024/09/13 - 13:06:38 | 200 | 14.255666264s |      172.17.0.1 | POST     "/api/chat"
check_double_bos_eos: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?
[GIN] 2024/09/13 - 13:06:41 | 200 |  2.408644743s |      172.17.0.1 | POST     "/api/chat"
check_double_bos_eos: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?
[GIN] 2024/09/13 - 13:07:09 | 200 |  8.941356956s |      172.17.0.1 | POST     "/api/chat"
check_double_bos_eos: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?
[GIN] 2024/09/13 - 13:07:45 | 200 | 10.676653407s |      172.17.0.1 | POST     "/api/generate"
check_double_bos_eos: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?
[GIN] 2024/09/13 - 13:07:48 | 200 |  2.450888118s |      172.17.0.1 | POST     "/api/generate"
check_double_bos_eos: Added a BOS token to the prompt as specified by the model but the prompt also starts with a BOS token. So now the final prompt starts with 2 BOS tokens. Are you sure this is what you want?

To Reproduce

  1. Install twinny
  2. Install ollama
  3. Pull deepseek-coder-v2:16b
  4. Configure providers in twinny (see screenshot below)
  5. Try using chat or FIM

image

Expected behavior It should work

Screenshots If applicable, add screenshots to help explain your problem.

Logging Rnable logging in the extension settings if not already enabled (you may need to restart vscode if you don't see logs). Proivide the log with the report.

API Provider ollama

Chat or Auto Complete? both chat and fim

Model Name Provide the model name you are using e.g codellama:7b-code or codellama:7b-instruct

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

rjmacarthy commented 1 week ago

Hello, I think this was my mistake by updating the default chat path to /api/chat when it should be /v1/chat/completions I just fixed it in the most recent version. For FIM the path /api/generate should be working.