FIM support for inline code completion

[jhgoebbert]

Just to ensure that FIM is on the radar at jupyter-ai I leave this comment here.

FIM (Fill-in-the-Middle, Fill-in-the-Middle) is to my knowledge a widely used approach in prompt-generation for inline code completion. Models are trained to understand a short list of keywords which describe where the "hole" in a text/code needs to be filled.

Google for example describes these keywords (for Gemma) here: https://ai.google.dev/gemma/docs/formatting#formatting-fim and shows an example here: https://ai.google.dev/gemma/docs/formatting#fim-example

The AI-code-assistence twinny uses the following FIM prompt generator for code-completion with the model StarCoder: getPrompt(..) -> getFimPrompt(..) -> getFimTemplateChosen(..) -> getFimPromptTemplateOther(..) -> return <fim_prefix>${fileContext}\n${heading}${prefix}<fim_suffix>${suffix}<fim_middle>

[JasonWeill]

This change may require an upstream change in JupyterLab's code completion extension. Opened https://github.com/jupyterlab/jupyterlab/issues/16671 for discussion at our weekly triage meeting tomorrow.

[jhgoebbert]

The tool llm.nvim also shows how to use FIM:

• CodeLlama uses the FIM tokens <PRE>, <MID>, <SUF>: https://github.com/huggingface/llm.nvim?tab=readme-ov-file#codellama
• Starcoder uses the FIM tokens <fim_prefix>, <fim_middle>, <fim_suffix>: https://github.com/huggingface/llm.nvim?tab=readme-ov-file#starcoder

DeepSeek-Coder also supports FIM with the tokens <｜fim▁begin｜>, <｜fim▁hole｜>, <｜fim▁end｜>. An example can be found here: https://github.com/deepseek-ai/DeepSeek-Coder?tab=readme-ov-file#2-code-insertion Comments of the model developer can be found here https://github.com/deepseek-ai/DeepSeek-Coder/issues/80

[krassowski]

It is already possible to swap relvant method in custom providers, in particular:

• the get_completion_prompt_template to adjust the prompt
• the generate_inline_completions and stream_inline_completions methods to circumvent the built-in prompt altogether and use a custom prompt machinery

See https://jupyter-ai.readthedocs.io/en/latest/developers/index.html#custom-completion-providers

I know that this has already been done in third-party providers (closed source). This only works within the constraints of the inline completion widget which is completing at the end of the line.

If you wish to have something that replaces a selection or generates code at the arbitrary cursor position in the code, we may need a different widget implementation or a revamp to the inline completer API. In that case suggestions for UX are welcome!

[jhgoebbert]

Hello @krassowski

thank you for the hints for a custom provider. This is great. I will try it :thumbsup: