Open vegansquirrel opened 9 months ago
The mixtral models have not been tested with lida/llmx.
I would recommend the following.
# spin up a vllm endpoint
from lida import Manager, TextGenerationConfig , llm
model_name = # mixtral model
model_details = [{'name': model_name, 'max_tokens': 2596, 'model': {'provider': 'openai', 'parameters': {'model': model_name}}}]
# assuming your vllm endpoint is running on localhost:8000
text_gen = llm(provider="openai", api_base="http://localhost:8000/v1", api_key="EMPTY", models=model_details)
lida = Manager(text_gen = text_gen)
On a somewhat related point, is there a way to use HuggingFace dedicated endpoint here?
While loading the model "ehartford/dolphin-2.5-mixtral-8x7b" into the Lida llm, I am getting the error shown in the image. I am loading it in a CPU. The llama models are loading fine but the new Mixtral is facing a problem.
I would highly appreciate a detailed response and solution.