search

ahyatt / llm

A package abstracting llm capabilities for emacs.
GNU General Public License v3.0
178 stars 24 forks source link

Ollama chat endpoint support #16

Closed tquartus closed 8 months ago

tquartus commented 8 months ago

Hello, Andrew,

First off, thanks for your project! I wrote some code over a year ago for interacting with the OpenAI API via Emacs. OpenAI broke that several times with their changes. Recently, I started working with LLMs locally and decided to rewrite everything from scratch to support multiple APIs ... and discovered that, thankfully, someone had beaten me to that! :-)

I noticed that the existing package provided support only for the Ollama /generate endpoint, but not the /chat endpoint. I find the /generate endpoint simpler for one-off requests. But for chat conversations, with multiple interactions, I prefer the /chat endpoint. For that, I can generate a set of interactions from scratch, regenerate responses, or switch between different LLMs in the same conversation. So, I've introduced some changes to support the other endpoint. I believe it's backward compatible.

The main changes include:

  1. A new :endpoint slot in the llm-ollama structure, which specifies the API endpoint to use. This slot defaults to generate, but can be set to chat to utilize the /chat endpoint.

  2. Adjustments to the JSON encodings for making requests to the API and processing responses from the server. These changes accommodate the differences between the /generate and /chat endpoints. Specifically, separate helper functions have been created to handle the request data for each endpoint.

  3. Modifications to the llm-ollama--chat-request function to call the appropriate helper function based on the specified endpoint.

  4. Updates to the llm-ollama--url calls in the llm-chat and llm-chat-streaming methods to use the specified endpoint.

  5. Updated documentation to reflect the new :endpoint slot in the llm-ollama structure.

Hopefully, support for both endpoints aligns with the package's goal of abstracting functionality to a higher level and concealing API variations.

I look forward to your feedback and the opportunity to contribute to the ongoing development of this package.

Best,
Thomas

ahyatt commented 8 months ago

Thanks for this change! Before I take a look, do you have FSF copyright assignment? This is needed to contribute substantial changes to this project, since it is part of GNU ELPA.

tquartus commented 8 months ago

In process. Emailed FSF just as I sent this in. Hopefully, won't take long. I'm a college professor, but our faculty handbook makes it clear they don't claim copyright over works by faculty. Hoping FSF will accept that without requiring additional signatures. Sorry for the delay. Will let you know when it's settled.

ahyatt commented 8 months ago

No problem, thank you for getting that process started! And without looking at your code, I'd suggest that it's OK to go to the chat endpoint by default. That is typically what we do for other providers.

tquartus commented 8 months ago

My FSF copyright assignment finally came through. Just wanted to let you know.

tquartus commented 8 months ago

Thanks! Based on your suggestions, I've removed support for the /generate endpoint and now only support the more versatile /chat endpoint. This simplifies the code and makes it more straightforward. As a result, the code involving string= and the :endpoint slot became unnecessary, so I've removed them as well, and also removed the added line from the README.org. Please let me know if there's anything else you'd like me to adjust.

ahyatt commented 8 months ago

Thank you for your change!