Bluefog-Lib / bluefog

Distributed and decentralized training framework for PyTorch over graph
https://bluefog-lib.github.io/bluefog/
Apache License 2.0
291 stars 71 forks source link

Check if bf.barrier() is working properly #51

Open hanbinhu opened 4 years ago

hanbinhu commented 4 years ago

Currently bf.barrier() doesn't make sure the order of the outputs is correct, for example the following code snippets

for i in range(2):
  bf.barrier()
  if rank == 0:
    print("A")
  bf.barrier()
  print("B")

Expected Output of bfrun -np 2

A
B
B
A
B
B

Possible output now:

A
B
B
B
A
B