search

Mintplex-Labs / anything-llm

The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, and more.
https://anythingllm.com
MIT License
26.53k stars 2.65k forks source link

[DOCS]: How to select your model in the program for Windows? #1039

Closed intulint closed 7 months ago

intulint commented 7 months ago

Description

I only found this instruction for docker. https://github.com/Mintplex-Labs/anything-llm/blob/master/server/storage/models/README.md#text-generation-llm-selection How to add your model to the program? Creating a downloaded folder in AppData\Roaming\anythingllm-desktop\storage\models\ doesn't work. Before this, I wanted to connect the model via koboldcpp, but the application does not support this backend. I need a model with q8, I don’t like ollama, but the worst thing is that it downloads 4q versions of neural networks. They're too stupid.

timothycarambat commented 7 months ago

If that is the case, then this issue needs to be worked #513 to get full support for you then

timothycarambat commented 7 months ago

Otherwise, you may certainly get a sub-par performance. Some people have said that using the LocalAI connector but pointed at your Kobold endpoint works? Oddly enough

intulint commented 7 months ago

@timothycarambat I tried every available local menu item, nothing wanted to work with kobaldcpp. Judging by the fact that that task has been hanging for 4 months, adding another local backend is of no interest to anyone. Which seems very strange. Since there is no llama.cpp server, no kobaldcpp, no text-generation-webui.

But there are several server part models that are cumbersome and not easy to launch, as well as adding the llm model you need with the launch parameters you need. impossible or inconvenient.

timothycarambat commented 7 months ago

@intulint It's been in the backlog because we don't use it and have only seen the request 2, maybe 3 times now. If we prioritize every single provider that gets added we will never move on past that set of features since there are an unlimited amount of tools for running LLMs. Everyone has a preference and it's unreasonable to expect us to add them instantly if the demand is not high enough. Looking now, that request has 4 votes on it.

That's just the way it is. Also, we are OSS, so if you really need it today, you can add it. This strategy we have taken thus far is not impossible or inconvenient for those who use the tool in its current state, just for you and others who need XYZ integration.

In an ideal world, every LLM provider is supported so nobody has any requests, but we are a small core team and contribution is optional, not required. So at the end of the day, we are responsible for doing most integrations.

Since that is what this issue is about - I'll close this issue and it can be tracked on #513. Until then.

intulint commented 7 months ago

@timothycarambat

But the problem for which I opened this ticket is not in the launch method and it has not yet been resolved. I don't agree with the closure.

I did not find how to add my already downloaded model to the program that is installed by the installer. This is not in the documentation, that's all. There is a mention that you can add your own model in the Docker version, but it is not clear how in the application.

I have a feeling that everyone is trying to make users depend on some servers from which models will be downloaded. Neither here, nor in lmstudio, nor most likely in ollama, you can simply add your pre-downloaded model. This, guys, already looks like some kind of conspiracy, heh.