search

facebookresearch / ClassyVision

An end-to-end PyTorch framework for image and video classification
https://classyvision.ai
MIT License
1.59k stars 277 forks source link

Enable NCCL_ASYNC_ERROR_HANDLING in Classy Vision #650

Closed osalpekar closed 3 years ago

osalpekar commented 3 years ago

Summary: NCCL Async Error Handling is a new mechanism implemented in ProcessGroupNCCL to provide reliability for DDP training runs using NCCL. See here for a more detailed background and implementation details: https://github.com/pytorch/pytorch/issues/46874.

At a high-level, this system was designed to ensure desynchronization, high GPU utilization, and NCCL errors don't cause indefinite hanging in distributed training runs. This system catches these errors without any perf impact and brings down the training process, and torchelastic can detect this and restart training from the previous checkpoint. The time after which stuck collectives are detected can be tuned using the timeout argument to init_process_group.

Differential Revision: D24840115

facebook-github-bot commented 3 years ago

This pull request was exported from Phabricator. Differential Revision: D24840115

facebook-github-bot commented 3 years ago

This pull request was exported from Phabricator. Differential Revision: D24840115

facebook-github-bot commented 3 years ago

This pull request has been merged in facebookresearch/ClassyVision@75a8e9501267bc13b42ea614211df454e7339e60.