AllentDan
commented
4 months ago https://lmdeploy.readthedocs.io/en/latest/supported_models/supported_models.html currently, settings except fp16 are not guaranteed.
Closed isaac-vidas closed 4 months ago
AllentDan
commented
4 months ago https://lmdeploy.readthedocs.io/en/latest/supported_models/supported_models.html currently, settings except fp16 are not guaranteed.
irexyc
commented
4 months ago Could you check if it will work if replacing this line as below ?
# with torch.device('meta'), warnings.catch_warnings():
from accelerate import init_empty_weights
with init_empty_weights(), warnings.catch_warnings():BTW, I don't think you need to install autoawq
isaac-vidas
commented
4 months ago Thanks @irexyc and @AllentDan!
I tried the change @irexyc suggested and it passed the previous part until it hits the following error:
$ python main.py
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [00:00<00:00, 3089.45it/s]
You have loaded an AWQ model on CPU and have a CUDA device available, make sure to set your model on a GPU device in order to run your model.
`low_cpu_mem_usage` was None, now set to True since model is quantized.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 6.11it/s]
Some weights of the model checkpoint at /home/gcpuser/sky_workdir/awq/llava-v1.5-13b-4bit were not used when initializing LlavaLlamaForCausalLM: ['model.mm_projector.0.weight', 'model.mm_projector.2.weight']
- This IS expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of LlavaLlamaForCausalLM were not initialized from the model checkpoint at /home/gcpuser/sky_workdir/awq/llava-v1.5-13b-4bit and are newly initialized: ['model.mm_projector.0.qweight', 'model.mm_projector.0.qzeros', 'model.mm_projector.0.scales']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
File "/home/gcpuser/sky_workdir/main.py", line 91, in <module>
main()
File "/home/gcpuser/sky_workdir/main.py", line 57, in main
pipe = pipeline(
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/lmdeploy/api.py", line 94, in pipeline
return pipeline_class(model_path,
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/lmdeploy/serve/vl_async_engine.py", line 16, in __init__
self.vl_encoder = ImageEncoder(model_path)
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/lmdeploy/vl/engine.py", line 68, in __init__
self.model = load_vl_model(model_path)
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/lmdeploy/vl/model/builder.py", line 36, in load_vl_model
return LlavaVisionModel(model_path)
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/lmdeploy/vl/model/llava.py", line 37, in __init__
self.build_model()
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/lmdeploy/vl/model/llava.py", line 69, in build_model
model.to(self.device).eval().half()
File "/opt/conda/envs/lmdeploy_env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2561, in half
raise ValueError(
ValueError: `.half()` is not supported for quantized model. Please use the model as it is, since the model has already been casted to the correct `dtype`.Could you provide the config.json file ?
isaac-vidas
commented
4 months ago This is the config for the quantized model:
{
"_name_or_path": "models/llava-v1.5-13b/",
"architectures": [
"LlavaLlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 1,
"eos_token_id": 2,
"fp16": true,
"freeze_mm_mlp_adapter": false,
"freeze_mm_vision_resampler": false,
"hidden_act": "silu",
"hidden_size": 5120,
"image_aspect_ratio": "pad",
"initializer_range": 0.02,
"intermediate_size": 13824,
"max_length": 4096,
"max_position_embeddings": 4096,
"mm_hidden_size": 1024,
"mm_patch_merge_type": "flat",
"mm_projector_lr": null,
"mm_projector_type": "mlp2x_gelu",
"mm_resampler_type": null,
"mm_use_im_patch_token": false,
"mm_use_im_start_end": false,
"mm_vision_select_feature": "patch",
"mm_vision_select_layer": -2,
"mm_vision_tower": "openai/clip-vit-large-patch14-336",
"model_type": "llava",
"num_attention_heads": 40,
"num_hidden_layers": 40,
"num_key_value_heads": 40,
"pad_token_id": 0,
"pretraining_tp": 1,
"quantization_config": {
"bits": 4,
"group_size": 128,
"quant_method": "awq",
"version": "gemm",
"zero_point": true
},
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"rope_theta": 10000.0,
"tie_word_embeddings": false,
"tokenizer_model_max_length": 4096,
"tokenizer_padding_side": "right",
"torch_dtype": "float16",
"transformers_version": "4.40.1",
"tune_mm_mlp_adapter": false,
"tune_mm_vision_resampler": false,
"unfreeze_mm_vision_tower": false,
"use_cache": false,
"use_mm_proj": true,
"vocab_size": 32000
}You can try if it will work after removing quantization_config in the config
"quantization_config": {
"bits": 4,
"group_size": 128,
"quant_method": "awq",
"version": "gemm",
"zero_point": true
},@ AllentDan will add better quantization support for vlm model next mounth.
isaac-vidas
commented
4 months ago It's working now 🎉 Thanks for all the help and this awesome library!
I'll add this hack until additional support will be added to the code.
Closing this for now.
zjysteven
commented
3 months ago @isaac-vidas Hello, may I ask what version of lmdeploy were you using when quantizing llava? I somehow encountered the problem of transformers not having llava model registered error, described here #1601.
Checklist
Describe the bug
Hi 👋, I quantized llava 1.5 13b model with the following command:
When trying to run the following code:
Encounter the following error:
Additional notes:
TheBloke/llava-v1.5-13B-AWQmodel with the same issue.awq.modules. I installedlmdeploy[all]and also manually installedautoawq. I tried with bothautoawq==0.2.4andautoawq==0.2.2and encountered the same error.accelerate/utils/modeling.pyismodel.mm_projector.0.qweightwhich is the first one.Did anyone else encounter this?
Reproduction
Described in the previous section
Environment
Error traceback