Open EatenBagpipe opened 6 months ago
You can set gss
to only zero. https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tests/kernels/weightOnly/weightOnlyKernelTest.cpp#L444
@Tracin Thanks for your reply. Within the actual behavior
section's initial batch of test scenarios, it seems that the condition for gss
being 0 is satisfied. But there is still a significant difference between the results of CudaKernel and CutlassKernel.
I found that the implementation of CudaKernel
by default assumes the layout for weight
is ColumnMajorTileInterleave
, whereas the CutlassKernel
used on sm70 devices utilize a ColumnMajor
layout. This difference leads to the inconsistency in the final calculation results.
Maybe the support for ColumnMajor
in the weightOnlyKernelTest
could be added, or just skip this test for sm70 devices.
@Barry-Delaney Hi Barry, Can you help with this? Thanks!
Hi @EatenBagpipe, thanks for the feedback! Sm70 support for weight only CUDA kernels is ongoing now, and will be in the next few code updates.
System Info
Who can help?
@Tracin
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Expected behavior
It's should pass all perChannel tests, and raise an expect when running finegraind quantization test.
actual behavior
additional notes
I compiled trt-llm from its source code on a V100 server and attempted to execute the WeightOnly unittest, but it resulted in a failure. Although I am aware that the Volta architecture does not support fine-grained weight-only quantization, I anticipated that the perChannel quantization would successfully pass the unittest as expected.