Open DrNochi opened 1 month ago
While I did not find the issue already being reported here, I did find a similar issue being reported with vLLM. They even just fixed it as well, so maybe their changes could be ported over to TGI?
Hi @DrNochi 👋
Thanks for reporting this, I'll tag @danieldk, he is a marlin expert!
Side note: Why is TGI even falling back on using Marlin kernels? As far as I know NVIDIA L4 is using the Ada Lovelace Architecture with CUDA compute capability 8.9, which should have hardware support for FP: NVIDIA CUDA Docs. Am I missing something?
Having a quick look through the code, I found the following PR https://github.com/huggingface/text-generation-inference/pull/2277 which was part of the latest release, basically "blocking" TGI from utilizing the native FP8 support by forcing the Marlin kernels for CC 8.9. I did not find any issues or further explaination pertaining to these changes. Maybe @OlivierDehaene could shed some light on the reasoning behind this change?
Side note: Why is TGI even falling back on using Marlin kernels? As far as I know NVIDIA L4 is using the Ada Lovelace Architecture with CUDA compute capability 8.9, which should have hardware support for FP: NVIDIA CUDA Docs. Am I missing something?
We switched to fbgemm-gpu
for FP8 matmul. However, it uses TMA (Tensor Memory Accelerator), which is not supported in CC 8.9.
System Info
Information
Tasks
Reproduction
To reproduce please run the following shell script:
The follow exception appears during startup:
Expected behavior
I expect no
RuntimeError
during shard initialization. TGI should start up and server the model without problems.