Hi! I find that you use "dist.all_gather_object" function to accomplish cross-GPU contrastive keys communication. But When I delete this part, the training time declines obviously. So I wonder if there is another more fancy way to break this time bottleneck? However I search for the pytorch distributed function and I haven't seen any function that can transport the tensor data in different shape except for "dist.all_gather_object", it seems that it is hard to obtain a tradeoff. Do you have any idea? Thanks!
Hi! I find that you use "dist.all_gather_object" function to accomplish cross-GPU contrastive keys communication. But When I delete this part, the training time declines obviously. So I wonder if there is another more fancy way to break this time bottleneck? However I search for the pytorch distributed function and I haven't seen any function that can transport the tensor data in different shape except for "dist.all_gather_object", it seems that it is hard to obtain a tradeoff. Do you have any idea? Thanks!