Open ultmaster opened 3 years ago
Hi, you could check this sshbarrier
config, which will start all tasks at the similar time
https://github.com/microsoft/pai/blob/9d7c1aca76269d61e36ab46feca1d667a64154e1/marketplace-v2/horovod-pytorch-synthetic-benchmark.yaml#L67-L72
@abuccts Thanks for reply. But what I want is to have a barrier at the middle of my task, after, for example, my data download is complete.
Is there any recommended practice to wait for preparation complete for all tasks? That is, to insert a barrier.
I've implemented one (for server-client scenario) with PyTorch, though I believe there might be better options. For example, something that has been natively supported by pai runtime.