search

art-from-the-machine / Mantella

Mantella is a Skyrim and Fallout 4 mod which allows you to naturally speak to NPCs using Whisper (speech-to-text), LLMs (text generation), and xVASynth / XTTS (text-to-speech).
https://art-from-the-machine.github.io/Mantella/
GNU Affero General Public License v3.0
169 stars 42 forks source link

Internal network LocalAI as LLM Provider #333

Closed maxi1134 closed 1 month ago

maxi1134 commented 1 month ago

Hi there, I currently use https://localai.io/ as as a basis for my LLM system.

I am wondering how hard it would be to integrate this with Mantella? This would reduce the load on my gaming PC using an AMD card and send the LLM load to my server with a 3090.

Thanks!

Leidtier commented 1 month ago

I haven't looked deeper into it, but if it really is a

Drop-in replacement for OpenAI

it provides an OpenAI compatible endpoint and you can already connect to it.

Just set the llm_api config value to the address of the endpoint provided by LocalAI.

maxi1134 commented 1 month ago

Just set the llm_api config value to the address of the endpoint provided by LocalAI.

That was it, thanks!