mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference
https://localai.io
MIT License
25.73k stars 1.93k forks source link

feature: rust https://github.com/rustformers backend #939

Open mudler opened 1 year ago

mudler commented 1 year ago

Is your feature request related to a problem? Please describe. A rust backend can be a robust alternative, and rustformers supports also other models

Describe the solution you'd like Have a rust backend to include support to models supported by rustformers

Describe alternatives you've considered

Additional context

For reference: https://github.com/huggingface/candle https://github.com/hyperium/tonic https://capnproto.org/

@lu-zero promised to give some help here =)

localai-bot commented 1 year ago

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

I can help you with your feature request for LocalAI. To clarify, you would like to have a rust backend added to LocalAI to support models that are supported by rustformers. This will provide an alternative to the existing C++ and Python backends. To implement this feature, we can start by adding the necessary dependencies and code to build and train the models using the rust backend. We can also ensure that the rust backend is compatible with the existing API and infrastructure of LocalAI. As for alternatives, we could consider other programming languages such as C++ or Python, but rust has several advantages over these languages in terms of performance and safety. Let me know if you have any further questions or if you would like to discuss this feature request further.

Sources:

lu-zero commented 1 year ago

Some notes:

Aisuko commented 1 year ago

Hi, guys. I am interested in this one. May I ask you guys the processing of this? I remember that we have an issue mentioned we need to have an example of building an external backend for localAI. I believe if we finish the example first and it would be more easy for us to do this. We need a template of external backend service.

@mudler May you please split the issue to a set of issues? We can pick up one by one. And it can also make sure we can do everything in the way that we want to.

mudler commented 1 year ago

we have https://github.com/go-skynet/LocalAI/issues/1057 already to track documentation on how to create LocalAI backends

lu-zero commented 1 year ago

Here is the equivalent tutorial but using tonic.

Aisuko commented 1 year ago

Noticed and thanks.

Aisuko commented 1 year ago

Here is an idea of the choosing of the default backend of rust #1219