This PR introduces veDeviceMesh, the device mesh API that integrates handling of submeshes and process groups in performing training with DDP, TP/SP, distributed optimizer and checkpointing. It also updates fixes and patches related to veDeviceMesh API to the repository since last PR.
This PR introduces veDeviceMesh, the device mesh API that integrates handling of submeshes and process groups in performing training with DDP, TP/SP, distributed optimizer and checkpointing. It also updates fixes and patches related to veDeviceMesh API to the repository since last PR.