Closed wanchaol closed 3 months ago
This PR is a follow up PR to enable fp8 allgather in TP after these PR landed:
One need to update their pytorch/float8_experimental to have those changes in to train with fp8 changes.
Since fp8 is not enabled as part of our integration tests yet, there should be no issues on CI or trains that does not use fp8
This PR is a follow up PR to enable fp8 allgather in TP after these PR landed:
One need to update their pytorch/float8_experimental to have those changes in to train with fp8 changes.
Since fp8 is not enabled as part of our integration tests yet, there should be no issues on CI or trains that does not use fp8