[Bug] AttributeError: 'LlavaNextConfig' object has no attribute 'hidden_size'

zhaozeno commented 6 days ago

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[ ] 2. The bug has not been fixed in the latest version.

Describe the bug

download the llava-v1.6-vicuna-7b-hf model to local folder, and run the command: lmdeploy serve api_server /data/models/huggingface/hub/llava-v1.6-vicuna-7b-hf

It reports error: llava-1 | 2024-06-27 11:10:15,226 - lmdeploy - WARNING - Try to run with pytorch engine because /data/models/huggingface/hub/llava-v1.6-vicuna-7b-hf is not explicitly supported by lmdeploy. llava-1 | 2024-06-27 11:10:22,420 - lmdeploy - ERROR - AttributeError: 'LlavaNextConfig' object has no attribute 'hidden_size' llava-1 | 2024-06-27 11:10:22,421 - lmdeploy - ERROR - test failed! llava-1 | ("Checking failed with error 'LlavaNextConfig' object has no attribute 'hidden_size'", 'Please send issue to LMDeploy with error logs.')

Reproduction

download the llava-v1.6-vicuna-7b-hf model to local folder, and run the command:

lmdeploy serve api_server /data/models/huggingface/hub/llava-v1.6-vicuna-7b-hf

Environment

Use latest docker image run the command.

Error traceback

No response

DefTruth commented 6 days ago

same as https://github.com/InternLM/lmdeploy/issues/1725

irexyc commented 6 days ago

The transformers version llava (LlavaNextConfig arch) is supported after release 0.4.2，so you can use it with lmdeploy 0.4.2.

You can install the nightly-build lmdeploy here https://github.com/zhyncs/lmdeploy-build/releases/tag/a06174f

InternLM / lmdeploy