Closed michaelthreet closed 1 month ago
Thanks for your issue @michaelthreet! Interestingly enough the error you get RuntimeError: [FT Error] Heurisitc failed to find a valid config.
seems to come from TRT-LLM: https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/tensorrt_llm/kernels/cutlass_kernels/cutlass_heuristic.cpp#L372
Maybe cc @Narsil or @mfuntowicz if you've seen this error before
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
System Info
OS version: Ubuntu 22.04 Model being used: Qwen/Qwen2-72B-Instruct Hardware being used: 4x 40GB A100 Deployment specificities: Running via docker using the
latest
tag as of 06/26Information
Tasks
Reproduction
Command was:
Expected behavior
A running LLM. The output I get is below: