Bluefog-Lib / bluefog

Distributed and decentralized training framework for PyTorch over graph
https://bluefog-lib.github.io/bluefog/
Apache License 2.0
291 stars 71 forks source link

Unifying weight definition #14

Closed hanbinhu closed 4 years ago

hanbinhu commented 4 years ago

The weight definition is required to be unified. Currently, we consider the weight should contain two parts: self_weight and neighbor_weights. self_weight serves as the weight for the self rank, and neighbor_weights serves as the weights for all the (in-)neighbors, which is consistent with the definition of neighbor in the topology. The following weight definition should be unified.

  1. win_sync
  2. weights inside topology
  3. MPI neighbor operation