search

DevXT-LLC / ezlocalai

ezlocalai is an easy to set up local artificial intelligence server with OpenAI Style Endpoints.
MIT License
66 stars 12 forks source link

Support different formats Huggingface phi 3 #43

Closed birdup000 closed 1 month ago

birdup000 commented 2 months ago

When trying to load the phi 3 models on HuggingFace it fails load since the format is different in the repo.

Josh-XT commented 2 months ago

Try changing DEFAULT_MODEL in your .env to this:

DEFAULT_MODEL=mradermacher/Phi-3-mini-128k-instruct-LLaMAfied-GGUF
birdup000 commented 2 months ago

Try changing DEFAULT_MODEL in your .env to this:

DEFAULT_MODEL=mradermacher/Phi-3-mini-128k-instruct-LLaMAfied-GGUF

Yes that one loads and works but the other ones do not work since its the format is different but its still GGUF

Josh-XT commented 2 months ago

Those following the standard TheBloke set on Hugging Face will work, but the people who have not followed those standards may have issues. I am planning to try to expand on that to try to find the file in whichever repo but I haven't yet. Currently it is limited to those who are in-standard with TheBloke's repos. I recommend looking at quants from https://huggingface.co/MaziyarPanahi