Open sfc-gh-zhwang opened 12 months ago
We have run into the same issue. Does anyone have any clue?
Hi @sfc-gh-zhwang , have you found a solution yet? I'm having the same issue here with running it on Kubernetes.
@Wenhan-Tan I just encountered the same issue. The reason I ran into this problem was that I had enabled hugepages on the physical machine, and UCX triggered a SIGBUS when trying to allocate memory using hugepages. Everything worked fine after I disabled hugepages.
@sphish Thank you! I saw another similar issue here (https://github.com/NVIDIA/TensorRT-LLM/issues/674) which uses TRT-LLM instead of FT. But in that issue, huge pages need be enabled. I'll try disabling huge pages first.
@sphish Thank you! I saw another similar issue here (NVIDIA/TensorRT-LLM#674) which uses TRT-LLM instead of FT. But in that issue, huge pages need be enabled. I'll try disabling huge pages first.
I think the key is that containers and bare metal need to have the same configuration
The same setup works fine on A100x8, but on H100x8, saw below errors.