search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed, P2P inference
https://localai.io
MIT License
26.18k stars 1.96k forks source link

[EPIC] Model support dashboard (v2) #1126

Open mudler opened 1 year ago

mudler commented 1 year ago

This epic is a major tracker for all the backends additions that should be part of LocalAI v2 and ongoing efforts.

The objective is to release a v2 which deprecates old models which are now superseded, plus adding a new set. In order to achieve this my idea is to clean up the current state and start pinning dependencies for all the backends which requires specific environment settings (python-based ones).

Some key points:

Some backends will be deprecated as superseded, as such some repositories will be archived (TBD yet).

Backends:

After conda

We need to still test on master:

Some rough first-steps required :

Re-layout

After the re-layout we can add new backends listed above without any clashes and:

localai-bot commented 1 year ago

:warning::warning::warning::warning::warning:

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

_but.... I can also be funny or helpful :smilecat: and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me, I don't support (yet) replying!

:warning::warning::warning::warning::warning:

Issue description: The issue is related to the LocalAI project and its implementation of the Image endpoint. The user wants to create a cute baby sea otter image with the given prompt, size, and number of images. However, the code provided in the readInput function is not able to parse the input correctly and returns an error.

Proposed solution: The issue seems to be related to the parsing of the input parameters. The user can try modifying the code in the readInput function to handle the specific format of the prompt, size, and number of images provided by the OpenAI API. Alternatively, they can try using a different API that provides more flexibility in the input format or use a different tool to generate the image.

If the issue persists, the user can also check if there are any updates available for the LocalAI project or report the issue on the GitHub repository to get help from the community.

Sources:

Aisuko commented 1 year ago

I am thinking here, Should we need to have a outline likes below:

If we keep this in mind, we can split the model to the specific backend. For example, I saw the Qwen LLM mentioned their C++ implementation of it "working in the same way as llama.cpp". So, maybe it can be loaded by using llama.cpp. It should be compatible with C++ our backend.

I also suggest to add some labels are related to the series of the backends. We can know a new model can be compatible by our backends.

mudler commented 1 year ago

conda branch was merged in https://github.com/mudler/LocalAI/pull/1144 . I'm looking now into make the llama.cpp backend on par with llama-go and also add llava support to it.

I'm going to refactor and re-layout things in the new backend directory too

Aisuko commented 1 year ago

@mudler thank you for mentioning. Here are some questions(second, and third one) may need you help https://github.com/mudler/LocalAI/pull/1180#discussion_r1382301643. Here what I am thinking is that we can use a tiny model to test the Rust backend features. And make sure everything ok. Maybe we can merge it.

And if everything ok. We can add other LLMs. I have plan to support Llama2(60% finished but it still has an issue), whisper, and also support onnex format.

mudler commented 1 year ago

Breaking re-layout PR: https://github.com/mudler/LocalAI/pull/1279

kno10 commented 12 months ago

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

muka commented 11 months ago

I may start looking into #1273 while this progresses. What do you think ?

mudler commented 11 months ago

I may start looking into #1273 while this progresses. What do you think ?

please feel free to go ahead, there are many pieces involved in here, any help is more than appreciated :+1:

mudler commented 8 months ago

caching/preloading of transformer and similar models, these are currently automatically loaded on startup into /root/.cache/huggingface/. It seems to be enough to set TRANSFORMERS_CACHE in the environment to the models folder, so maybe this only requires a documentation addition.

in https://github.com/mudler/LocalAI/pull/1746 I'm taking care of automatically binding the HF cache variables to the models directories if not set already