Can't get the FIM to work with LM Studio

[Wyzix33]

Describe the bug I've tried to use WizardCoder for FIM and it's not working, here is the log for LM Studio:

[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 11434
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] ->  GET  http://localhost:11434/v1/models
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] ->  POST http://localhost:11434/v1/chat/completions
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] ->  POST http://localhost:11434/v1/completions
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] ->  POST http://localhost:11434/v1/embeddings     <------------ NEW!
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Model loaded: mike-ravkine/WizardCoder-15B-V1.0-GGUF/WizardCoder-15B-V1.0.Q5_1.gguf
[2024-04-21 11:39:29.795] [INFO] [LM STUDIO SERVER] Logs are saved into C:\tmp\lmstudio-server-log.txt

[2024-04-21 11:41:23.856] [INFO] [LM STUDIO SERVER] Processing queued request...
[2024-04-21 11:41:23.857] [INFO] Received POST request to /v1/completions with body: {
  "model": "WizardCoder",
  "prompt": "<PRE> /**/\n\n/* Language: PHP (php) */\n/* File uri: file:///c%3A/github/magura/php/nomenclator/parteneri/clienti.php (php) */\n $sql = \"SELECT * FROM clienti\";\n$result = $conn->query($sql);\n$clienti = [];\nwhile ($row = $result->fetch_assoc()) {\n $clienti[] = $row;\n}\n$sql = \"SELECT * FROM clienti_sedii\";\n$result = $conn->query($sql);\n$sedii = [];\nwhile ($row = $result->fetch_assoc()) {\n $sedii[] = $row;\n}\nheader(\"Content-Type: application/json\");\necho json_encode(['clienti' => $clienti, 'sedii' => <SUF> \n <MID>",
  "stream": true,
  "temperature": 0.2,
  "n_predict": 512
}
[2024-04-21 11:41:23.858] [INFO] Provided inference configuration: {
  "temp": 0.2
}
[2024-04-21 11:41:23.859] [INFO] Streaming response..
[2024-04-21 11:41:23.860] [INFO] [LM STUDIO SERVER] Processing...
[2024-04-21 11:41:24.520] [INFO] [LM STUDIO SERVER] Finished streaming response

To Reproduce start a server with any coder LLM and test, sometimes i get a completion but it's nonsense...

Expected behavior A completion based on the context... in my case it should fill "$sedii]);" but sometimes i get nothing, other times i get some "#import <...>"

Desktop (please complete the following information):

• OS: Windows
• Version 11

Additional context I used twinny v3.11.9 and VSCode Version: 1.88.1 (user setup) Commit: e170252f762678dec6ca2cc69aba1570769a5d39 Date: 2024-04-10T17:41:02.734Z Electron: 28.2.8 ElectronBuildId: 27744544 Chromium: 120.0.6099.291 Node.js: 18.18.2 V8: 12.0.267.19-electron.0 OS: Windows_NT x64 10.0.22631

Chat seems to work tho... Thanks

[rjmacarthy]

Hello, it seems you are using WizardCoder for FIM but with codellama template. I don't know which fim template WizardCoder supports or if it supports FIM. You should try a different model. Either deepseek base or codellama-code.

https://github.com/rjmacarthy/twinny/blob/main/docs/providers.md#fim-auto-complete-2

I see here: https://huggingface.co/epinnock/wizardcoder-1b-merged#fill-in-the-middle that WizardCoder uses the same FIM template as Starcoder, please try that as the FIM template, or use custom and edit your own.

See docs for recommended settings.

[Wyzix33]

thanks for the answer, I've made progress but still not good enough, at least now i can get the FIM to work, here is what i get, I will try other LLM maybe the ones i got are not good...

https://github.com/rjmacarthy/twinny/assets/13553412/21210c7c-bb5b-4351-9183-780d03703d09

[rjmacarthy]

Hey, starcoder2 has an issue with completions, I have not been successful with it either. Please try codellama or deepseek-coder. Many thanks.