Mantella is a Skyrim and Fallout 4 mod which allows you to naturally speak to NPCs using Whisper (speech-to-text), LLMs (text generation), and xVASynth / XTTS (text-to-speech).
Create an interface that provides the conversation logic with a unified simple access to the different things we want an LLM to generate
Spoken sentences
Switching Character
Narrations
Actions
Vision analysis
etc.
Create different implementations to this interface that then in turn provide the requested things (or not if unable)
By separating creation and consumption clearly we can develop them independently (like creating the support in the game, even if the LLMs are still triggering it unreliable)
Different implementation of these can make us of
Specific implementations for different LLM services (OpenAI, OpenRouter, text-gen-webui, koboldcpp, other)
Different implementations for ways to get structured output out of LLMs (JSON schema, Tools, function calls, text parsing)
Goals
Progress