search

dotnet-smartcomponents / smartcomponents

Experimental, end-to-end AI features for .NET apps
622 stars 54 forks source link

Support for Ollama API would be great #17

Closed cheindel closed 3 months ago

cheindel commented 3 months ago

Ollama has an API that is compatible to the OpenAI API and lets me run LLMs locally. However, if I change "Endpoint" in configuration to the appropriate local Ollama URL it does not work. Please consider supporting Ollama, as due to the nature of the data being processed, I cannot use OpenAI in the cloud.

SteveSandersonMS commented 3 months ago

We can definitely do this. In fact by coincidence I was doing it myself yesterday.

One caveat though, is that the llama2:7b output is not very good for this, and probably not good enough to be usable in reality. I didn't try the bigger Llama2 models - perhaps they are better. But Llama2 7B's output was pretty weird on Smart Paste and Smart Textarea, compared with great quality results from gpt3.5-turbo (and gpt3.5-turbo is about 10x faster too).

I'd be very interested to know if you find bigger Llama2 models behave better. More encouragingly, I did get quite good output from Mistral 7B (albeit still too slow on my dev machine's GPU).

cheindel commented 3 months ago

Give me the connectivity and I can try out different LLMs. 🤝

SteveSandersonMS commented 3 months ago

OK, this feature is now available. Just update your Smart Components packages to version 0.1.0-preview10146 or later, and then see config instructions at https://github.com/dotnet-smartcomponents/smartcomponents/blob/main/docs/configure-openai-backend.md#using-ollama-or-another-self-hosted-endpoint.

I'll again caveat that many models won't produce good or usable output, so please don't judge Smart Components as a whole based on whatever experience you get with the first local models you try. I haven't yet found a local model that works well, but it would be great if one did exist and we could add it to the recommendations (so please let us know if you do!).

cheindel commented 3 months ago

At a first glance, autocompletion in German language using Sroecker/Sauerkrautlm-7b-Hero:Latest looks good, running fast enough to be usable on a MacBook Pro M1 Max with 64 GB RAM. I will run some more tests including "Smart Paste" next week. Thank you for this initiative and and the fast response.