[FR] Add kwargs to `idist.barrier`

[vfdev-5]

🚀 Feature

A feature request to add kwargs to idist.barrier.

• Verify if all supported backends now can accept additional args to their barrier methods
• Add kwargs and route it to backend implementations
  • warn if kwargs are specified but can not be used by the backend
• Update documentation
• Add test

[sdesrozis]

Note :

• https://github.com/horovod/horovod/blob/master/horovod/torch/mpi_ops.py#L203
• https://github.com/pytorch/xla/blob/master/torch_xla/core/xla_model.py#L861

The parameters name for horovod and tag (replicas ?) for xla could be defined.

[fco-dv]

I can try to tackle this one!

[fco-dv]

@sdesrozis for horovod : https://github.com/horovod/horovod/blob/66ad6d5a3586decdac356e8ec95c204990bbc3d6/horovod/torch/mpi_ops.py#L974 seems to have added a barrier method, could it be a replacement of the workaround implemented here : https://github.com/pytorch/ignite/blob/4a37e35dcb9548bb777dc86ab649ff390c6fb631/ignite/distributed/comp_models/horovod.py#L197 ?

[sdesrozis]

@fco-dv Yes but it's pretty recent, so we should handle previous horovod version.