search

fra31 / auto-attack

Code relative to "Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks"
https://arxiv.org/abs/2003.01690
MIT License
639 stars 111 forks source link

Parallelized computing #94

Closed Bai-YT closed 2 years ago

Bai-YT commented 2 years ago

Hi,

I am trying to run AutoAttack on multiple GPUs to accelerate the experiments. However, it seems that simply wrapping the model with torch.nn.DataParallel will result in one GPU running at nearly 100% with other GPUs only at about 20%. So I just hoped to ask if there is a more efficient way of parallelization. Has there been a DistributedDataParallel code example for AutoAttack? Thank you very much.

fra31 commented 2 years ago

Hi,

I haven't tried myself, but if I remember right @max-andr used AA with parallelization at some point, maybe he has some insights.

max-andr commented 2 years ago

Hi,

actually, I haven't tried it myself, so can't really tell, sorry :/

Bai-YT commented 2 years ago

Thank you @max-andr @fra31 for the information!

CNOCycle commented 1 year ago

In my forked repo, torch.nn.DataParallel works properly. When the evaluation is run on two GPUs, it reduces the elapsed time by about half. Please share your code to further investigation.

Running auto-attack with DistributedDataParallel is much complex. Some control flows should be refined to retrieve intermediate information efficiently or avoid deadlocks. Those mechanisms or programming logic highly depends how you launch DDP version of auto-attack. However, if all you need is the final robust accuracy, about 20 lines modifications are required.

Bai-YT commented 1 year ago

In my forked repo, torch.nn.DataParallel works properly. When the evaluation is run on two GPUs, it reduces the elapsed time by about half. Please share your code to further investigation.

Running auto-attack with DistributedDataParallel is much complex. Some control flows should be refined to retrieve intermediate information efficiently or avoid deadlocks. Those mechanisms or programming logic highly depends how you launch DDP version of auto-attack. However, if all you need is the final robust accuracy, about 20 lines modifications are required.

Thank you for sharing your experience! Much appreciated.