When building the docker images from scratch on a google cloud instance, I ran into an issue with the pytorch version installed by pip not being compatible with TensorRT-LLM. Following this issue thread https://github.com/NVIDIA/TensorRT-LLM/issues/808 I was able to resole the issue by introducing these version pins to the respective Docker build scripts.
When building the docker images from scratch on a google cloud instance, I ran into an issue with the pytorch version installed by pip not being compatible with TensorRT-LLM. Following this issue thread https://github.com/NVIDIA/TensorRT-LLM/issues/808 I was able to resole the issue by introducing these version pins to the respective Docker build scripts.