search

s-kostyaev / ellama

Ellama is a tool for interacting with large language models from Emacs.
GNU General Public License v3.0
348 stars 25 forks source link

[PR?] Fill in the middle completer #113

Open NightMachinery opened 2 months ago

NightMachinery commented 2 months ago

I have written a usable fill-in-the-middle function. The code is available here (look for night/ellama-code-fill-in-the-middle), and I can clean it up further for inclusion in ellama, if there's interest. Or someone else can do that.

Here are some demos with llama3-70b-8192:

https://github.com/s-kostyaev/ellama/assets/36224762/ac7dcf6d-c3cd-423e-aa6e-57540b037417

https://github.com/s-kostyaev/ellama/assets/36224762/71b007fb-04e3-46b2-acb8-a7905c44437d

The highlights will be cleared the next time you press C-g, though this depends on Doom's doom-escape-hook; I couldn't find a built-in hook for keyboard-quit.

s-kostyaev commented 2 months ago

Hi @NightMachinery Looks interesting. Have you signed FSF papers? If so, open PR 🙂

NightMachinery commented 2 months ago

I have signed the copyright papers for emacs before; are these the same?

I'll use this function for a while to see how robust it is. Its implementation is very hacky currently, and the heuristics used look somewhat model-dependent.

s-kostyaev commented 2 months ago

I have signed the copyright papers for emacs before; are these the same?

Yes. It means you don't need to do it second time.

I'll use this function for a while to see how robust it is. Its implementation is very hacky currently, and the heuristics used look somewhat model-dependent.

we can improve it during code review. And it will be model dependent, because different models handle FIM differently, has different tags etc.

NightMachinery commented 2 months ago

different models handle FIM differently

The current implementation is a hack that uses normal LLMs tuned for chat. If the model supported FIM, the implementation would become a lot simpler. I am not aware of any FIM API though. (Surely there must be some open models, but is there anything competent?)

s-kostyaev commented 2 months ago

See codeqwen, deepseek coder (not latest one) and Starcoder 2 7b (in 15b fim is broken).

NightMachinery commented 2 months ago

See codeqwen, deepseek coder (not latest one) and Starcoder 2 7b (in 15b fim is broken).

Is there any cloud API for these models? (e.g., on OpenRouter or OpenAI)

I think the llm backend doesn't support FIM completion, either.

s-kostyaev commented 2 months ago

I don't think llm should do something to add fim support, it's about right prompt format. I don't know about cloud APIs, I don't use it.