Decide on initial local LLM server option

[ryderwishart]

• [ ] Must be able to run on < 12gb RAM

A smaller model could be leveraged for few-shot, single-token or ngram translation suggestions (e.g., 2-3B params). This is the simplest use case and something we might want to leverage first.

A larger model will be needed for chat, and we need some way to figure out which model the user needs based on their source language settings. This is obviously somewhat fragile, but we can begin to support various gateway languages.

We can actually fire this up through txtai right now to keep things simple, though we will also want to use PredictionGuard for an API version, so when we get to more consistently online users we will likely want a proxy that can gracefully navigate between both.

[ryderwishart]

Hindi

• Hindi instruction-tuned LLM

Telugu

• abhinand/telugu-llama-7b-instruct-v0.1
• eswardivi/llama2chat_telugu
• PosteriorAI/godavari-telugu-llama2-7B

Kannada

• jayavibhav/Llama-2-Kannada

Tamil

• Lots of models here

[ryderwishart]

For now we opted to simply use LM Studio. We can reopen this later if we want to mask this from the user too, but it would take a lot of work to rival LM Studio's flexibility at this point.