how does the backward msg get the corresponding old weights in 1F1B-RR

gbxu commented 4 years ago

hi, me again.. First of all, your work is really good, so I read it once more time.

For "1F1B-RR" mechanism, let's assume that there is 1 machine(A) in previous stage and 2 machines(B, C) in the current stage. It's a simple 1-2 configuration, and pipeline the data with in-flight=2.
A process forward of batch3, backward of batch2, the forward of batch4. Then A send batch3 to B and batch4 to C. I want to know what'll happen if C finish the stage then send the backward of batch4 to A earlier than B's batch3. I find that load_old_params() seems to just pop the buff queue. So the batch4 would get the wrong weights which doesn't include the gradients of batch2.
I'm not sure that the functionality of setup_messaging_schedule. Is setup_messaging_schedule function designed for this corner case?

deepakn94 commented 4 years ago

The activations / gradients processed by PipeDream are explicitly controlled in the runtime: receive queues are read in a deterministic order, which means PipeDream is resistant to messages from stage replicas arriving in different orders. As you pointed out, lines 586 and 592 implement this logic.

gbxu commented 4 years ago

thx, deepak!

msr-fiddle / pipedream

how does the backward msg get the corresponding old weights in 1F1B-RR #22