search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference
https://localai.io
MIT License
26.92k stars 2.01k forks source link

feature: support CTranslate2 #839

Open mudler opened 1 year ago

mudler commented 1 year ago

Among the backends that we already have, we should consider also ctransformers2 which is high performance: https://opennmt.net/CTranslate2/generation.html besides #796

localai-bot commented 1 year ago

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

Great suggestion! We can definitely consider adding support for ctransformers2 as it seems to be a powerful backend. I will add this to our short-term roadmap and we will investigate the feasibility of implementing it. Thank you for bringing this up!

Sources:

Aisuko commented 1 year ago

@mudler Show more detail please. Do you mean we use it on Text generation or encode?