search

c0sogi / llama-api

An OpenAI-like LLaMA inference API
MIT License
111 stars 9 forks source link

BUG: I found the model path bug! #17

Closed Dougie777 closed 1 year ago

Dougie777 commented 1 year ago

So this has been driving me crazy. I thought I was losing my mind. So I finally figured it out.

In my model definitions I had:

WizardLM_70B_q4_GGUF = LlamaCppModel( model_path="wizardlm-70b-v1.0.Q4_K_M.gguf", # manual download max_total_tokens=4096, use_mlock=False, )

but when I listed the model definitions in the API I got:

{
  "id": "wizardlm_70b_q4_gguf",
  "object": "model",
  "owned_by": "LlamaCppModel",
  "permissions": [
    "model_path:wizardlm-70b-v1.0.Q4_K_M.gguf",

......

It converted the model id to lower case!!!!!!!!!! So I changed my model definition to be all lower case AND IT WORKS!

So to fix either we need to clearly document that model definitions variable names MUST be in lower case. Or change the code to not convert to lower case.

** But this is not the whole story. I have a working model definition with upper case letters working... So something I am saying is not correct. But the above procedure definitely fixed my problem.

c0sogi commented 1 year ago

Yes you got the right point. I lower-cased all the model names because I wanted it to work case-insensitively. This was a mistake on my part, and I modified the code to handle the same models regardless of case (upper or lower case).

I slightly modified my code to work this out: 6b254fdaab2ac2337e6b93d910b41a96f8de2a80 Now, both wizardlm_70b_q4_gguf and wizardLM_70b_q4_GGuf will work.

Dougie777 commented 1 year ago

Thanks so much !!