search

dimagi / open-chat-studio

A web based platform for building Chatbots backed by Large Language Models
BSD 3-Clause "New" or "Revised" License
15 stars 7 forks source link

Open Source & locally hosted LLMs #292

Open ronna opened 7 months ago

ronna commented 7 months ago

Is it possible to use open source and locally hosted LLMs? I dont have access to comercial hosted LLMs due to 1. not affording to pay for them, 2. not having an acceptable payment method and 3. the commercial LLMs offering not vailable in my country (Zimbabwe)

bderenzi commented 7 months ago

Not currently, though it is definitely on our roadmap to support open source LLMs. Locally hosted LLMs might be slightly lower priority, though I could imagine a version where there's a lightweight API for that open source, local-hosted LLM and we can just point to that API.

snopoke commented 7 months ago

This should be relatively easy to enable by adding OpenLLM as a provider type in Open Chat Studio.

As a point of reference here's the PR where Anthropic was added: https://github.com/dimagi/open-chat-studio/pull/212 which an almost identical process.

bderenzi commented 7 months ago

Oh, great. Yeah, that looks exactly like what I had in mind as a little lightweight API to run locally to serve the locally-hosted LLMs in a consistent way