Ollama support

[roman]

Hello @ahyatt,

Thank you for contributing the code from this project. I'm looking forward to use it as it seems the one with the least amount of OpenAI coupling I've seen.

I'm interested in running this extension with an ollama backend. This tool is pretty neat as it allow clients to run various OSS data models in a straightforward way. The tool runs a local HTTP server that provides an API to query the various models. It also offers a CLI to interact with the various model installations, model tweaks, etc.

I've the impression that implementing a backend for this shouldn't be as challenging as you have lots of foundations going on already. I'm happy to help if there is some work that can be split out.

[roman]

This short overview video is also very informative.

[ahyatt]

Thanks, I hadn't heard of ollama. Is the code to interface with ollama something you are interesting in contributing? And, if so, do you have FSF papers? Otherwise, it seems like something I can work on, but I also have several other next steps and don't know yet how I'll prioritize things.

[roman]

Is the code to interface with ollama something you are interesting in contributing?

I can give it a stab at the work, I'm playing with ways to integrate emacs to local LLMs so this will probably be my first attempt at it.

do you have FSF papers?

I don't think so, what are those?

Otherwise, it seems like something I can work on, but I also have several other next steps and don't know yet how I'll prioritize things.

That's reasonable, I was not expecting this to get prioritized right away.

Thank you for your work.

[roman]

Hey @ahyatt,

I can run the llm-test.el file using ert no problem. However, I'm unsure how to run llm-tester for the various backends (OpenAI, Vertex, etc).

Are there some steps you follow to test that?

[ahyatt]

Yes, for that I create a provider, and run llm-tester-all on it, and look in the Messages buffer for the output. For example:

(llm-tester-all (make-llm-openai :key my-openai-key))

For the FSF paperwork, if you wanted to pursue contributing, you need to fill this out and follow the instructions. Let me know soon if you intend to do that, otherwise, I took a look at the ollama API and it should be fairly trivial for me to do this.

[roman]

For the FSF paperwork, if you wanted to pursue contributing, you need to fill this out and follow the instructions. Let me know soon if you intend to do that

I filled the form and submit it a couple of minutes ago

I took a look at the ollama API and it should be fairly trivial for me to do this.

If you feel empowered to do this work, please don't wait for me. I'm more than happy to review it 👍

[ahyatt]

I've added an ollama provider today, @roman can you try it out and make sure it works for you? Also, please check the embeddings and make sure they make sense - are similar concepts closer in embedding space?

[roman]

I've added an ollama provider today, @roman can you try it out and make sure it works for you?

It is working 👍 , thanks!

Also, please check the embeddings and make sure they make sense - are similar concepts closer in embedding space?

To be quite honest with you, I'm new to the LLM game, and I'm not acquainted yet with the concept of embeddings or how to use them, let me get some knowledge about that and get back to you

One Nit, what do you think about allowing ollama to be served from another host that is not localhost? I can see a setup where a company runs an ollama instance for various employees and you can use that instead.

[ahyatt]

About non-localhost serving: sure, I can do that, but is it possible? IIRC, localhost serving often will forbid outside connections. So I guess it depends on if ollama is designed to be used that way.

[roman]

but is it possible? IIRC, localhost serving often will forbid outside connections. So I guess it depends on if ollama is designed to be used that way.

I think you can setup ollama to bind on 0.0.0.0 to allow remote connections. I can imagine a more sophisticated setup (https, authentication, etc.) being built on top of that.

[roman]

By the way, is there any project that sits on top of this project to integrate with org babel blocks, or have a chat buffer?

I have the impression the goal of this project is to be nimble and serve as a lower tier for more sophisticated integrations with the editor.

[ahyatt]

One of my other projects, ekg uses this, but it's for a pretty specialized use. There are a number of other emacs Chat GPT integrations and I've started contacting them to use this library. Hopefully they will switch soon - it would make sense for them, since they already are getting requests to use different backends, and it isn't great for them to have to each figure out how to make the calls and handle streaming.

[roman]

Sounds good,

I'm going to close this ticket as completed now.

Thanks for the quick turnaround about this @ahyatt your work and efforts are highly appreciated