[Bug]: portkey_ai.api_resources.exceptions.NotFoundError: Error code: 404 - Unable to find requested model. Please confirm that the model exists and you have permission. (Request ID: )

[pradeepdev-1995]

Contact Details

No response

What happened?

I tried to prompt to the mistralai/Mistral-7B-Instruct-v0.2 model by Anyscale virtual key(Without adding the model in anyscale account). It shows the above error.

So how to use the opensource models directly from the huggingface or the fientuned models from my local folder?

Version

0.1.xx (Default)

Relevant log output

No response

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[vrushankportkey]

Hi @pradeepdev-1995 thanks for reporting. Can you please share the error you're getting while calling the model on Anyscale? Portkey currently doesn't fully support HuggingFace OR local models. That said, support for Ollama local models is coming soon! https://github.com/Portkey-AI/gateway/pull/194

[pradeepdev-1995]

So at present it supports only openai/cohere/or open source models via Anyscale models? I can't use the opensource models directly from the huggingface or the fientuned models from my local folder.? @vrushankportkey

[vrushankportkey]

This is the exhaustive list of providers Portkey supports:

Open source models are available through Anyscale, Together, Perplexity, Mistral endpoints, with more coming soon.

HuggingFace as well as local models support is in progress.

Hope that helps!

[pradeepdev-1995]

Thank you