How to arrange for DistributedDataParallel

NUS-HPC-AI-Lab / Neural-Network-Parameter-Diffusion

We introduce a novel approach for parameter generation, named neural network parameter diffusion (p-diff), which employs a standard latent diffusion model to synthesize a new set of parameters

828 stars 42 forks source link

How to arrange for DistributedDataParallel #18

Closed liuyifan22 closed 5 months ago

liuyifan22 commented 5 months ago

Hello! I am inspired by your great work. I found that it might be beneficial to use DDP, but it does not seem to have been implemented in your code. I tried to use DDP or PytorchLightning, but both seem to get stuck in the progress: DDP: stuck in the line DDP(system) Lightning: stuck in "[rank: 0] Seed set to 42 Initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2"

Would you please help me out by telling me how to DDP? Thanks!

liuyifan22 commented 5 months ago

Sorry, it is actually a problem with A6000. Already solved, and I'm looking forward to your official DDP version!