Without having dist.barrier(), all of the HF examples wind up hanging since we're destroying the pg before all comms have completed in these small examples, leading to a hang. This PR adds dist.barrier() just before dist.destroy_process_group() to fix this.
Without having
dist.barrier(), all of the HF examples wind up hanging since we're destroying the pg before all comms have completed in these small examples, leading to a hang. This PR addsdist.barrier()just beforedist.destroy_process_group()to fix this.