Open rosario-purple opened 7 months ago
We haven't test MS-AMP with pytorch 22. Currently we only support pytorch1.14 and 2.1. And it is recommended to use our docker image or nvcr.io/nvidia/pytorch:23.10-py3. And we have plan to upgrade msccl to latest version.
Can you share me the complete steps of reproducing this issue?
What's the issue, what's expected?:
I tried to compile the MS-AMP optimizer with the new Torch 2.2:
but got this error:
How to reproduce it?:
Running this code in Python reproduces the error:
Log message or shapshot?:
See above
Additional information:
My best guess is that this is caused by MS-AMP being pinned to an external old version of libnccl (2.17.1), while PyTorch 2.2 seems to depend on a newer version (2.19.3).