Open Aktsvigun opened 5 months ago
byshiue
commented
5 months ago For tokens <= 4, we use cuda kernel and it should work on all GPUs. But for tokens > 4, we use cutlass to leverage the tensor core. Since we use some new features only supported when SM >= 80 in cutlass, so the kernel cannot be used on T4.
Aktsvigun
commented
5 months ago Thank you! I can then reformulate the question: are there any options to run ModelRunnerCpp on T4? Maybe with different quantization scheme etc. ?
byshiue
commented
5 months ago For GPTQ with batch_size * max_input_length > 4, we use cutlass to accelerate and some features are only supported when SM >= 80. So, TensorRT-LLM does not support GPTQ on T4 now.
Aktsvigun
commented
5 months ago Hi @byshiue , do I understand it right that T4 can't handle AWQ as well? It produces the following error when using trying to quantize with AWQ (unsure whether I need to create a special topic for this):
Traceback (most recent call last):
File "/srv/storage/al/repo/TensorRT-LLM/examples/llama/../quantization/quantize.py", line 376, in <module>
main(args)
File "/srv/storage/al/repo/TensorRT-LLM/examples/llama/../quantization/quantize.py", line 283, in main
model = quantize_model(model, quant_cfg, calib_dataloader)
File "/srv/storage/al/repo/TensorRT-LLM/examples/llama/../quantization/quantize.py", line 220, in quantize_model
atq.quantize(model, quant_cfg, forward_loop=calibrate_loop)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/ammo/torch/quantization/model_quant.py", line 112, in quantize
calibrate(model, config["algorithm"], forward_loop=forward_loop)
File "ammo/torch/quantization/model_calib.py", line 59, in ammo.torch.quantization.model_calib.calibrate
File "ammo/torch/quantization/model_calib.py", line 185, in ammo.torch.quantization.model_calib.awq
File "ammo/torch/quantization/model_calib.py", line 187, in ammo.torch.quantization.model_calib.awq
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "ammo/torch/quantization/model_calib.py", line 330, in ammo.torch.quantization.model_calib.awq_lite
File "/srv/storage/al/repo/TensorRT-LLM/examples/llama/../quantization/quantize.py", line 216, in calibrate_loop
model(data)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1044, in forward
outputs = self.model(
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 929, in forward
layer_outputs = decoder_layer(
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 654, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
output = module._old_forward(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 255, in forward
query_states = self.q_proj(hidden_states)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/llm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "ammo/torch/quantization/model_calib.py", line 294, in ammo.torch.quantization.model_calib.awq_lite.forward
NotImplementedError: Cannot copy out of meta tensor; no data!All the system parameters are the same.
byshiue
commented
5 months ago @Aktsvigun T4 does not support AWQ. But the error you report should not be related to T4. If you still have issue, you could create another issue to describe the issue you encounter.
Aktsvigun
commented
5 months ago Thanks @byshiue ! Are there any quantization schemes that T4 does support?
byshiue
commented
5 months ago Since T4 is not officially supported by TensorRT-LLM now, we cannot guarantee which case T4 could support. You could try int8 weight only or SQ.
System Info
(1) nvidia-smi
(2) nvcc
(3) pip list
Who can help?
@byshiue
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
Note: the first 3 steps are taken from official examples.
(1) Quantize with GPTQ:
python llama.py PATH_TO_SAVED_HF_MODEL c4 --wbits 4 --true-sequential --groupsize 128 --save_safetensors PATH_TO_SAVE_QUANTIZED_MODEL/MODEL_NAME.safetensors(2) Convert checkpoint:
python convert_checkpoint.py --model_dir PATH_TO_SAVED_HF_MODEL --output_dir TLLM_OUTPUT_DIR --dtype float16 --ammo_quant_ckpt_path PATH_TO_SAVE_QUANTIZED_MODEL/MODEL_NAME.safetensors --use_weight_only --weight_only_precision int4_gptq --per_group(3) Build:
trtllm-build --checkpoint_dir TLLM_OUTPUT_DIR --output_dir OUTPUT_DIR --gemm_plugin float16(4) Run with ModelRunnerCpp:
Expected behavior
When running this pipeline on A100, everything is normally generated.
actual behavior
When running this on T4 as in the system info above, kernel dies in jupyter notebook; running a python script results in:
additional notes
I saw in other threads many problems related to running on T4, and I guess in my example something is not compatible with T4 as well. Yet, I cannot understand which part needs to be modified to tackle this issue (or maybe this is impossible).
Another interesting observation is that it works on shorter sequences (i.e. when input ids contain <= 4 tokens). Still this error is very unlikely to be related to OOM issue (I did monitor the CPU memory and it did not explode).