TheBloke--Llama-2-70B-chat-GPTQ model: weight model.layers.0.self_attn.q_proj.weight does not exist

ray-project / ray-llm

RayLLM - LLMs on Ray

https://aviary.anyscale.com

Apache License 2.0

1.22k stars 89 forks source link

TheBloke--Llama-2-70B-chat-GPTQ model: weight model.layers.0.self_attn.q_proj.weight does not exist #52

Closed YQ-Wang closed 1 year ago

YQ-Wang commented 1 year ago

I tried to serve TheBloke--Llama-2-70B-chat-GPTQ model with Aviary 0.2.0 and have the following error: RuntimeError: weight model.layers.0.self_attn.q_proj.weight does not exist. This seems to be an issue https://github.com/huggingface/text-generation-inference/issues/500 with TGI.

I noticed that the issue in Aviary to run TheBloke--Llama-2-70B-chat-GPTQ model is marked as resolved. Can you share any suggestions?

cc @Yard1

YQ-Wang commented 1 year ago

I was able to unblock myself by taking reference from this commit: https://github.com/ray-project/aviary/commit/2569f2d9e50ee27e4828d205ac9d75820c738da9#diff-aad8e2d49d52b24d24c8cdb142d7088c240998b37187401f0ea7fec0791ccd90. Close this issue for now. Thanks!