search

huanzhang12 / CLEVER

CLEVER (Cross-Lipschitz Extreme Value for nEtwork Robustness) is a robustness metric for deep neural networks
https://openreview.net/pdf?id=BkUHlMZ0b
Apache License 2.0
60 stars 20 forks source link

mnist_brelu&cifar_brelu models are missing #8

Open chengwuxinlin opened 3 years ago

chengwuxinlin commented 3 years ago

Hello huanzhang,

I saw in Built-in Models, there is a brelu" (7-layer CNN with Bounded ReLU) option. but brelu is missing inside the train network codes(train_models.py; train_2layer.py; train_nlayer.py). Could you please offer pre-trained mnist_brelu&cifar_brelu networks? Thanks

huanzhang12 commented 3 years ago

@chengwuxinlin Unfortunately I am not able to find these models now. But it should be easy to modify the training script a little bit to train them (just change activation function and model name).

Additionally, bounded ReLU is not a reliable way to enhance model robustness, so it is not recommended to use it anymore. For a more reliable way to improve robustness, you should look into adversarial training (e.g., Madry's work or TRADES) or certified defense (e.g., CROWN-IBP).

chengwuxinlin commented 3 years ago

Got it! Thanks

chengwuxinlin commented 3 years ago

@huanzhang12
Hi huanzhang, so in the compute clever score section, you used the average score among all chose data. But why not use the least score? the robustness level should be decided by the most vulnerable point right?

huanzhang12 commented 3 years ago

@chengwuxinlin You are right that on a single image, the robustness level is decided by the most vulnerable point. But the average is done over the dataset level. Because for each image the model has different level of robustness, we have to use some metric like the average to report a statistic. Mean is also used in other papers like the CW attack paper, Table 3.

chengwuxinlin commented 3 years ago

Also I saw in collect_gradients.py, it says: parser.add_argument("--ids", default = "", help = "use a filelist of image IDs in CSV file for attack (UNSUPPORTED)")

is this means I cannot use a file list of IDs for attack?