continuedev / continue

⏩ Continue is the leading open-source AI code assistant. You can connect any models and any context to build custom autocomplete and chat experiences inside VS Code and JetBrains
https://docs.continue.dev/
Apache License 2.0
19.43k stars 1.69k forks source link

Tweak Granite template and postprocessing #3025

Closed fbricon closed 7 hours ago

fbricon commented 19 hours ago

Description

This PR aims to improve granite3 models tab completion support. Granite3 is not FIM trained (unlike previous versions), so needs to use the holeFillerTemplate in AutocompleteTemplate.getTemplateForModel, instead of the default stableCodeFimTemplate. Additionally, we found that smaller variants of granite3 models tend to sometimes return results that do not always satisfy the hole filler prompt instructions, typically repeating existing code in the response. Granite specific post-processing was added to postprocessCompletion to try to mitigate those misbehaviors.

Finally, I chose to remove extra lines prepended to this.writeLog(Completion:` statements, as I was having difficulty visually parsing the actual model output. Now we can really tell if extra lines were returned by the model.

Checklist

Screenshots

Testing

The end of the prefix is then removed from the completion, so it can be inserted properly

netlify[bot] commented 19 hours ago

Deploy Preview for continuedev failed. Why did it fail? →

Name Link
Latest commit 3934e8d81facccc42a44932311a4f36906aec8df
Latest deploy log https://app.netlify.com/sites/continuedev/deploys/673f6ba24844a800089ee20f