search

IBM / CLEVER-Robustness-Score

Codes for reproducing the robustness evaluation scores in “Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach,” ICLR 2018 ​​​​​​​
https://openreview.net/pdf?id=BkUHlMZ0b
Apache License 2.0
47 stars 18 forks source link

Pytorch model #6

Open kfarivar opened 3 years ago

kfarivar commented 3 years ago

Hi I wanted to know how I can calculate the clever score for a pytorch model ?

pinyuchen commented 3 years ago

Hello @kfarivar You can use the general wrapper from IBM Adversarial Robustness Toolbox: https://adversarial-robustness-toolbox.readthedocs.io/en/stable/modules/metrics.html

huanzhang12 commented 3 years ago

Thank you @pinyuchen for the answer. Let us know if you have any further questions.

kfarivar commented 3 years ago

I have seen the wrapper. It has 2 major problems. 1. The implementation is not efficient. The original implementation uses multitasking library while theirs is a serial implementation.

  1. The implementation is incomplete. The original code estimates the c_init while that's not implemented in the wrapper.
huanzhang12 commented 3 years ago

@kfarivar I agree with you that the original implementation is more efficient. I did a lot of multiprocessing optimizations to fully utilize the GPU. The new implementation was done by an independent team which focuses more on functionality rather than performance.

It is possible to change the original implementation to use a PyTorch model. CLEVER is a sampling based method so as long as you can integrate pytorch model inference into the code, everything else should work. More precisely, you will need to replace this tensorflow sess.run with a pytorch model inference:

https://github.com/huanzhang12/CLEVER/blob/master/estimate_gradient_norm.py#L461

Most other things can be kept unchanged, including the multitasking part. However it does need some efforts if you plan to take this route.

liuyiming199721 commented 1 year ago

Hello, is there an easy way to use your pytorch models?