Support Mistral-7B-Instruct-v0.3

[WxxShirley]

Hi,

I have used fastchat to deploy Mistral-7B-Instruct-v0.2 as a local LLM service. However, when it comes to the recent 0.3 version, errors are invoked as follows:

2024-07-24 16:57:05 | ERROR | stderr |   File "/root/miniconda3/lib/python3.8/site-packages/vllm/model_executor/model_loader.py", line 103, in get_model
2024-07-24 16:57:05 | ERROR | stderr |     model.load_weights(model_config.model, model_config.download_dir,
2024-07-24 16:57:05 | ERROR | stderr |   File "/root/miniconda3/lib/python3.8/site-packages/vllm/model_executor/models/mistral.py", line 388, in load_weights
2024-07-24 16:57:05 | ERROR | stderr |     param = state_dict[name]
2024-07-24 16:57:05 | ERROR | stderr | KeyError: 'layers.0.attention.wk.weight'

• Command python -m fastchat.serve.vllm_worker --model-path mistralai/Mistral-7B-Instruct-v0.3 --host 127.0.0.1
• Packages vllm==0.2.1 fschat==0.2.36 transformers==4.42.4

If I have used the correct packages, can you fix the above bug and support the integration with Mistral-7B-Instruct-v0.3?

I have also tried Mistral-7B-v0.3, the same error happened.

[surak]

you need to update the versions of the underlying packages and it does run - I have it running on my premises.

I don't know how you install fastchat, but I use a git clone of the main branch, constantly updated.

It already works, so I will close this one, if that's okay for you. Please let me know if there's something else it should be done!

[WxxShirley]

Thank you so much for your reply!

Can you specify the versions of your installed packages, like fastchat and vllm? I have followed your instructions but it didn't work. Thank you once again for your kindness and time : )

[surak]

Hi!

I literally used the latest versions of sglang[srt], accelerate and vllm. Fastchat is, as I said, from git - the pip version is too old for latest models.

[WxxShirley]

Get it! Thank you so much for your information 😄