search

InternLM / lmdeploy

LMDeploy is a toolkit for compressing, deploying, and serving LLMs.
https://lmdeploy.readthedocs.io/en/latest/
Apache License 2.0
4.17k stars 376 forks source link

[Bug] Can't Quantize llava-v1.6-34b (AssertionError) #2079

Open rtadewald opened 1 month ago

rtadewald commented 1 month ago

Checklist

Describe the bug

Hello Guys. I'm having trouble trying to quantize the llava-v1.6-34b model here, following this tutorial.

I've successfully quantized smaller models (liuhaotian/llava-v1.6-vicuna-7b and liuhaotian/llava-v1.6-vicuna-13b) but when I try with the 34B, got the following error:

Reproduction

export HF_MODEL=liuhaotian/llava-v1.6-34b export WORK_DIR=quantized_vlms/llava-v1.6-34b-4bit lmdeploy lite auto_awq $HF_MODEL --work-dir $WORK_DIR

Environment

sys.platform: linux
Python: 3.10.14 (main, May  6 2024, 19:42:50) [GCC 11.2.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0,1: NVIDIA GeForce RTX 3090
CUDA_HOME: /usr
NVCC: Cuda compilation tools, release 12.0, V12.0.140
GCC: gcc (Ubuntu 13.2.0-23ubuntu4) 13.2.0
PyTorch: 2.2.2+cu121
PyTorch compiling details: PyTorch built with:
  - GCC 9.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.3.2 (Git Hash 2dc95a2ad0841e29db8b22fbccaf3e5da7992b01)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.5
  - Built with CUDA Runtime 12.1
  - NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
  - CuDNN 8.9.2
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.2.2, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 

TorchVision: 0.17.2+cu121
LMDeploy: 0.5.1+
transformers: 4.40.0
gradio: 3.35.2
fastapi: 0.111.0
pydantic: 2.8.2
triton: 2.2.0
NVIDIA Topology: 
        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-7,32-39       0               N/A
GPU1    SYS      X      8-15,40-47      2               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Error traceback

export HF_MODEL=liuhaotian/llava-v1.6-34b                                                                                                                                py lmdeploy rtadewald@ 23:25:09
export WORK_DIR=quantized_vlms/llava-v1.6-34b-4bit                            
lmdeploy lite auto_awq $HF_MODEL --work-dir $WORK_DIR 
can't find model from local_path liuhaotian/llava-v1.6-34b, try to download from remote
Fetching 23 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 6520.82it/s]
/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/transformers/models/llava/configuration_llava.py:103: FutureWarning: The `vocab_size` argument is deprecated and will be removed in v4.42, since it can be inferred from the `text_config`. Passing this argument has no effect
  warnings.warn(
2024-07-18 23:26:17,140 - lmdeploy - INFO - matching vision model: LlavaVisionModel
You are using a model of type llava to instantiate a model of type llava_llama. This is not supported for all configurations of models and can yield errors.
2024-07-18 23:26:17,833 - lmdeploy - INFO - CLIPVisionTower.load_model: openai/clip-vit-large-patch14-336
2024-07-18 23:26:18,218 - lmdeploy - INFO - CLIPVisionTower.load_model: openai/clip-vit-large-patch14-336
/home/rtadewald/.local/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Move model.embed_tokens to GPU.
Move model.layers.0 to CPU.
Move model.layers.1 to CPU.
Move model.layers.2 to CPU.
Move model.layers.3 to CPU.
Move model.layers.4 to CPU.
Move model.layers.5 to CPU.
Move model.layers.6 to CPU.
Move model.layers.7 to CPU.
Move model.layers.8 to CPU.
Move model.layers.9 to CPU.
Move model.layers.10 to CPU.
Move model.layers.11 to CPU.
Move model.layers.12 to CPU.
Move model.layers.13 to CPU.
Move model.layers.14 to CPU.
Move model.layers.15 to CPU.
Move model.layers.16 to CPU.
Move model.layers.17 to CPU.
Move model.layers.18 to CPU.
Move model.layers.19 to CPU.
Move model.layers.20 to CPU.
Move model.layers.21 to CPU.
Move model.layers.22 to CPU.
Move model.layers.23 to CPU.
Move model.layers.24 to CPU.
Move model.layers.25 to CPU.
Move model.layers.26 to CPU.
Move model.layers.27 to CPU.
Move model.layers.28 to CPU.
Move model.layers.29 to CPU.
Move model.layers.30 to CPU.
Move model.layers.31 to CPU.
Move model.layers.32 to CPU.
Move model.layers.33 to CPU.
Move model.layers.34 to CPU.
Move model.layers.35 to CPU.
Move model.layers.36 to CPU.
Move model.layers.37 to CPU.
Move model.layers.38 to CPU.
Move model.layers.39 to CPU.
Move model.layers.40 to CPU.
Move model.layers.41 to CPU.
Move model.layers.42 to CPU.
Move model.layers.43 to CPU.
Move model.layers.44 to CPU.
Move model.layers.45 to CPU.
Move model.layers.46 to CPU.
Move model.layers.47 to CPU.
Move model.layers.48 to CPU.
Move model.layers.49 to CPU.
Move model.layers.50 to CPU.
Move model.layers.51 to CPU.
Move model.layers.52 to CPU.
Move model.layers.53 to CPU.
Move model.layers.54 to CPU.
Move model.layers.55 to CPU.
Move model.layers.56 to CPU.
Move model.layers.57 to CPU.
Move model.layers.58 to CPU.
Move model.layers.59 to CPU.
Move model.norm to GPU.
Move model.vision_tower to GPU.
Move model.mm_projector to GPU.
Move lm_head to CPU.
Loading calibrate dataset ...
/home/rtadewald/.local/lib/python3.10/site-packages/datasets/load.py:1429: FutureWarning: The repository for ptb_text_only contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/ptb_text_only
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
/home/rtadewald/.local/lib/python3.10/site-packages/datasets/load.py:1429: FutureWarning: The repository for ptb_text_only contains custom code which must be executed to correctly load the dataset. You can inspect the repository content at https://hf.co/datasets/ptb_text_only
You can avoid this message in future by passing the argument `trust_remote_code=True`.
Passing `trust_remote_code=True` will be mandatory to load this dataset from the next major release of `datasets`.
  warnings.warn(
Token indices sequence length is longer than the specified maximum sequence length for this model (1190301 > 4096). Running this sequence through the model will result in indexing errors
model.layers.0, samples: 128, max gpu memory: 13.08 GB
model.layers.1, samples: 128, max gpu memory: 16.58 GB
model.layers.2, samples: 128, max gpu memory: 16.58 GB
model.layers.3, samples: 128, max gpu memory: 16.58 GB
model.layers.4, samples: 128, max gpu memory: 16.58 GB
model.layers.5, samples: 128, max gpu memory: 16.58 GB
model.layers.6, samples: 128, max gpu memory: 16.58 GB
model.layers.7, samples: 128, max gpu memory: 16.58 GB
model.layers.8, samples: 128, max gpu memory: 16.58 GB
model.layers.9, samples: 128, max gpu memory: 16.58 GB
model.layers.10, samples: 128, max gpu memory: 16.58 GB
model.layers.11, samples: 128, max gpu memory: 16.58 GB
model.layers.12, samples: 128, max gpu memory: 16.58 GB
model.layers.13, samples: 128, max gpu memory: 16.58 GB
model.layers.14, samples: 128, max gpu memory: 16.58 GB
model.layers.15, samples: 128, max gpu memory: 16.58 GB
model.layers.16, samples: 128, max gpu memory: 16.58 GB
model.layers.17, samples: 128, max gpu memory: 16.58 GB
model.layers.18, samples: 128, max gpu memory: 16.58 GB
model.layers.19, samples: 128, max gpu memory: 16.58 GB
model.layers.20, samples: 128, max gpu memory: 16.58 GB
model.layers.21, samples: 128, max gpu memory: 16.58 GB
model.layers.22, samples: 128, max gpu memory: 16.58 GB
model.layers.23, samples: 128, max gpu memory: 16.58 GB
model.layers.24, samples: 128, max gpu memory: 16.58 GB
model.layers.25, samples: 128, max gpu memory: 16.58 GB
model.layers.26, samples: 128, max gpu memory: 16.58 GB
model.layers.27, samples: 128, max gpu memory: 16.58 GB
model.layers.28, samples: 128, max gpu memory: 16.58 GB
model.layers.29, samples: 128, max gpu memory: 16.58 GB
model.layers.30, samples: 128, max gpu memory: 16.58 GB
model.layers.31, samples: 128, max gpu memory: 16.58 GB
model.layers.32, samples: 128, max gpu memory: 16.58 GB
model.layers.33, samples: 128, max gpu memory: 16.58 GB
model.layers.34, samples: 128, max gpu memory: 16.58 GB
model.layers.35, samples: 128, max gpu memory: 16.58 GB
model.layers.36, samples: 128, max gpu memory: 16.58 GB
model.layers.37, samples: 128, max gpu memory: 16.58 GB
model.layers.38, samples: 128, max gpu memory: 16.58 GB
model.layers.39, samples: 128, max gpu memory: 16.58 GB
model.layers.40, samples: 128, max gpu memory: 16.58 GB
model.layers.41, samples: 128, max gpu memory: 16.58 GB
model.layers.42, samples: 128, max gpu memory: 16.58 GB
model.layers.43, samples: 128, max gpu memory: 16.58 GB
model.layers.44, samples: 128, max gpu memory: 16.58 GB
model.layers.45, samples: 128, max gpu memory: 16.58 GB
model.layers.46, samples: 128, max gpu memory: 16.58 GB
model.layers.47, samples: 128, max gpu memory: 16.58 GB
model.layers.48, samples: 128, max gpu memory: 16.58 GB
model.layers.49, samples: 128, max gpu memory: 16.58 GB
model.layers.50, samples: 128, max gpu memory: 16.58 GB
model.layers.51, samples: 128, max gpu memory: 16.58 GB
model.layers.52, samples: 128, max gpu memory: 16.58 GB
model.layers.53, samples: 128, max gpu memory: 16.58 GB
model.layers.54, samples: 128, max gpu memory: 16.58 GB
model.layers.55, samples: 128, max gpu memory: 16.58 GB
model.layers.56, samples: 128, max gpu memory: 16.58 GB
model.layers.57, samples: 128, max gpu memory: 16.58 GB
model.layers.58, samples: 128, max gpu memory: 16.58 GB
model.layers.59, samples: 128, max gpu memory: 16.58 GB
model.layers.0 smooth weight done.
model.layers.1 smooth weight done.
model.layers.2 smooth weight done.
model.layers.3 smooth weight done.
model.layers.4 smooth weight done.
model.layers.5 smooth weight done.
model.layers.6 smooth weight done.
model.layers.7 smooth weight done.
model.layers.8 smooth weight done.
model.layers.9 smooth weight done.
model.layers.10 smooth weight done.
model.layers.11 smooth weight done.
model.layers.12 smooth weight done.
model.layers.13 smooth weight done.
model.layers.14 smooth weight done.
model.layers.15 smooth weight done.
model.layers.16 smooth weight done.
model.layers.17 smooth weight done.
model.layers.18 smooth weight done.
model.layers.19 smooth weight done.
model.layers.20 smooth weight done.
model.layers.21 smooth weight done.
model.layers.22 smooth weight done.
Traceback (most recent call last):
  File "/home/rtadewald/miniconda3/envs/lmdeploy/bin/lmdeploy", line 8, in <module>
    sys.exit(run())
  File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/cli/entrypoint.py", line 36, in run
    args.run(args)
  File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/cli/lite.py", line 139, in auto_awq
    auto_awq(**kwargs)
  File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/lite/apis/auto_awq.py", line 131, in auto_awq
    smooth_layers(layers, fc2fcs, norm2fcs, act_scales, w_group_size,
  File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/lite/quantization/awq.py", line 259, in smooth_layers
    smooth_ln_fcs(ln, fcs, a_scales[a_name], group_size)
  File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/lite/quantization/awq.py", line 118, in smooth_ln_fcs
    assert torch.isnan(p).sum() == 0
AssertionError
AllentDan commented 1 month ago

Two possible solutions:

  1. adjust your quantization settings such as sample nums or calibration dataset
  2. modify files following https://github.com/InternLM/lmdeploy/pull/1844/files
rtadewald commented 1 month ago

Hello, @AllentDan, thanks for your answer.

Unfortunately none of the proposed solutions worked for me. This getting the Assertion Error here:

File "/home/rtadewald/miniconda3/envs/lmdeploy/lib/python3.10/site-packages/lmdeploy/lite/quantization/awq.py", line 118, in smooth_ln_fcs assert torch.isnan(p).sum() == 0 AssertionError

AllentDan commented 1 month ago

Sorry for the late replay, I can do awq normally through:

lite auto_awq llava-v1.6-34b --work-dir llava-v1.6-34b-awq --calib-seqlen 512 --calib-dataset pileval
AllentDan commented 1 month ago

The NAN value was from act_scale, which I suppose it might be related to a calibration prompt that can generate a NAN value inside the layer.