search

cleverhans-lab / cleverhans

An adversarial example library for constructing attacks, building defenses, and benchmarking both
MIT License
6.17k stars 1.39k forks source link

Run time getting a lot slower #910

Closed B-Yassine closed 5 years ago

B-Yassine commented 5 years ago

Dear all, I just upgraded to cleverhans 2.1.0 version and it seems like the attacks are getting a lot slower for instance a Pgd attack was taking only 10 mn to generate the adversarial image but now it is taking about an hour and a half to generate it and I could not downgrade to 2.0.0 as it doesn't exist anymore in pypi

Is there an explanation for this problem, is it a problem with the tensorflow's graph generation?

I am using tensorflow version : 1.12.0 Cleverhans version: 2.1.0 Python version: 3.6

goodfeli commented 5 years ago

You can always download by manually cloning the early github tag.

I don't actually know about pypi. @npapernot should the old version actually be removed from pypi?

@B-Yassine : this kind of slowdown isn't expected. Are you sure you're using the same model and the same configuration for the attack (especially the number of iterations) as before? Has the attack moved from running on GPU to running on CPU?

npapernot commented 5 years ago

I responded to @B-Yassine's email and just saw this. For everyone else reading this: the old version is still in pypi:

B-Yassine commented 5 years ago

Yes, the old version 2.1.0 is still in pypi and you can get it simply by running: pip3 install cleverhans==2.1.0

Thank you for your responds

Best

On Mon, Nov 26, 2018, 9:53 PM Nicolas Papernot notifications@github.com wrote:

I responded to @B-Yassine https://github.com/B-Yassine's email and just saw this. For everyone else reading this: the old version is still in pypi:

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/cleverhans/issues/910#issuecomment-441794157, or mute the thread https://github.com/notifications/unsubscribe-auth/AoK_-iR-ZGvDhFlyAWebQVhFWkWK5gCzks5uzFTmgaJpZM4YxoJM .

goodfeli commented 5 years ago

Do you still want us to help debug the slowness in v3?

B-Yassine commented 5 years ago

Yes, of course because am very interested in this domain as I am preparing my Master thesis and being able to contribute will be a honor for me.

I think there's another problem with the deepfool attack as I tried there source code implemented in Pytorch and using the cleverhans implementation with the same parameters and the same model "inception v3" and the results where differents !

The author told me that he doesn't know about the cleverhans implementation and adviced me to use its own implementation

Best, Yassine

On Mon, Nov 26, 2018, 10:10 PM Ian Goodfellow notifications@github.com wrote:

Do you still want us to help debug the slowness in v3?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/cleverhans/issues/910#issuecomment-441799099, or mute the thread https://github.com/notifications/unsubscribe-auth/AoK_-psCWXMxsQNtpbzY3Gk15yuQtCIsks5uzFjQgaJpZM4YxoJM .

B-Yassine commented 5 years ago

For the deepfool attack i think it has to be opened as another issue

goodfeli commented 5 years ago

Yes, please open another issue with more detail for the deepfool attack.

goodfeli commented 5 years ago

Do you still want to try to debug the slowness issue?

B-Yassine commented 5 years ago

Thanks a lot so the problem has been resolved by downgrading to the previous version 2.1.0 and it can be done by running: pip install cleverhans==2.1.0 and making sure it is running on gpu