Open AndersGiovanni opened 7 months ago
Which exact settings of the GEN_AI_
variables did you try? For me the following works with a self-hosted Huggingface TGI:
GEN_AI_MODEL_VERSION="<huggingface-model-id>"
GEN_AI_MODEL_PROVIDER="huggingface"
HUGGINGFACE_API_BASE="https://xyz"
GEN_AI_API_ENDPOINT="https://xyz"
Disclaimer: I am unsure whether you need to set both HUGGINGFACE_API_BASE
and GEN_AI_API_ENDPOINT
but for me it works that way.
I'm using the following settings:
GEN_AI_MODEL_PROVIDER=huggingface
GEN_AI_MODEL_VERSION=mistralai/Mistral-7B-Instruct-v0.2
GEN_AI_API_KEY=some_key
HUGGINGFACE_API_BASE=https://api-inference.huggingface.co/models/
GEN_AI_API_ENDPOINT=https://api-inference.huggingface.co/models/
I also tried with other models like meta-llama/Llama-2-70b-chat-hf
and meta-llama/Llama-2-7b-chat-hf
.
I'm trying to replace the use of OpenAI with Llama from Huggingface with Inference Pro (I follow the instructions specified in https://docs.danswer.dev/gen_ai_configs/huggingface). The API key is specified in the .env. However when trying to chat, I get the following error:
Whenever I change the model to OpenAI and provide an API keys from them, everything works like a charm.
Any suggestions on how to fix this?
This is the log from the api-server when using huggingface.