Closed Bai-YT closed 2 years ago
Hi,
I haven't tried myself, but if I remember right @max-andr used AA with parallelization at some point, maybe he has some insights.
Hi,
actually, I haven't tried it myself, so can't really tell, sorry :/
Thank you @max-andr @fra31 for the information!
In my forked repo, torch.nn.DataParallel
works properly. When the evaluation is run on two GPUs, it reduces the elapsed time by about half. Please share your code to further investigation.
Running auto-attack with DistributedDataParallel
is much complex. Some control flows should be refined to retrieve intermediate information efficiently or avoid deadlocks. Those mechanisms or programming logic highly depends how you launch DDP version of auto-attack. However, if all you need is the final robust accuracy, about 20 lines modifications are required.
In my forked repo,
torch.nn.DataParallel
works properly. When the evaluation is run on two GPUs, it reduces the elapsed time by about half. Please share your code to further investigation.Running auto-attack with
DistributedDataParallel
is much complex. Some control flows should be refined to retrieve intermediate information efficiently or avoid deadlocks. Those mechanisms or programming logic highly depends how you launch DDP version of auto-attack. However, if all you need is the final robust accuracy, about 20 lines modifications are required.
Thank you for sharing your experience! Much appreciated.
Hi,
I am trying to run
AutoAttack
on multiple GPUs to accelerate the experiments. However, it seems that simply wrapping the model withtorch.nn.DataParallel
will result in one GPU running at nearly 100% with other GPUs only at about 20%. So I just hoped to ask if there is a more efficient way of parallelization. Has there been aDistributedDataParallel
code example forAutoAttack
? Thank you very much.