Account for FIM tokens in prompt

huggingface / llm-ls

LSP server leveraging LLMs for code completion (and more?)

Apache License 2.0

602 stars 49 forks source link

Account for FIM tokens in prompt #62

Closed HennerM closed 8 months ago

HennerM commented 8 months ago

For the Fill-in-the-middle prompt we need extra tokens to designate start, middle and end. This is usually one token each, so need to subtract 3 token of the max context length

McPatate commented 8 months ago

Hi @HennerM, thanks a lot for all your contributions. I'll take a look when I have some time, I'm currently busy on other subjects. Stay put!