search

matthewwicker / CertifiableBayesianInference

Repository for performing Certifiably Robust Bayesian inference for deep neural networks
5 stars 0 forks source link

Dear Dr.Wicker, Is there any code for drawing pictures in the paper? #2

Closed Hongchenglong closed 2 years ago

Hongchenglong commented 2 years ago

Dear Dr.Wicker, Is there any code for drawing pictures in the paper?

an9236868 commented 2 years ago

I agree. It's really hard to reproduce the paper's results with the current state of the repo. A simple instruction would be really appreciated.

Hongchenglong commented 2 years ago

Dear Dr.Wicker, I have successfully started the code, but I did not find the code for drawing. Are these diagrams generated by code? I want these codes. Thank you very much. image

matthewwicker commented 2 years ago

Hi Both,

Thank you for your interest in the paper and in the code to reproduce the results. I am sorry you have found the project hard to run. Some simple instructions for now:

runner.sh - runs the training. Command: $./runner.sh analysis.py - runs the analysis. Command: $python3 analysis.py

I see how a confusing part would be my encoding of the training modes. Run with "rob" option 0 to get normal training "rob" option 1 to get IBP training, and "rob" option 2 to get PGD training. The codes for the "opt" option are the same as the optimizer values listed in the paper.

I have not put the code for the figures in this repo. They are simply matplotlib bar plots.

I will update the repo with some more detailed instructions, but will try to investigate what your issues are first so I know what level of detail to go into.

Best, Matthew Wicker

matthewwicker commented 2 years ago

I agree. It's really hard to reproduce the paper's results with the current state of the repo. A simple instruction would be really appreciated.

Following up from my main post in this thread, can you please tell me what you have tried up to this point and what specifically you found confusing?

Hongchenglong commented 2 years ago

Looking forward to your more detailed instructions about this experiment. Thank you.

matthewwicker commented 2 years ago

Sure, I will write out exactly how you do this with the repository and if you run into errors let me know.

Let's take the top row of the figure. First thing to do is train the BNNs. To do this navigate to the 'FCN_Experiments' directory. From there you can either use the ./runner.sh command to train all of the BNNs in parallel or you can use the MNIST_runner.py to train a specific BNN. Let's discuss the later as it gives you more control than the former. Running the MNIST_runner.py can be done with the following line:

python3 MNIST_runner.py --eps 0.11 --lam 0.25 --rob 0 --opt VOGN

This line has four key parameters of interest. First is the epsilon (--eps), this determines the strength of the adversary and corresponds to the \eta in equation 5 of the paper. The second is is lambda (lam) this corresponds to \lambda in equation 5 of the paper and determines the probability of seeing the clean data (lambda = 1.0 means no adversarial data considered, lambda = 0.0 means only adversarial data considered). Next is the robustness mode flag (rob), as mentioned previously this controls if you are training normally (--rob 0) i.e. clean data only, training with IBP adversary (--rob 1) or a PGD adversary (--rob 2). Lastly is the variational inference method you would like to learn with and this is flagged with the opt command. Simply pass the capital lettered code on the figure y-axis to select one.

Now that we have covered how to train a network, let us discuss the analysis. The training code will write all of the posterior information to a directory. By supplying the opt and rob flag to analysis.py (e.g. python3 analysis.py --opt VOGN --rob 0) the file will output the accuracy, robustness to attacks, and certified robustness.

One can then log these values and use matplotlib bar plots to display the data as in the figure in the paper.

I think this is about as detailed an explanation as one might need, but please do let me know if you require any further details. I am happy to discuss, consult, and collaborate on extensions of this project! Please just send me an email with the particulars and I will get back to you.

Best, Matthew Wicker

Hongchenglong commented 2 years ago

Dear Dr.Wicker, my email is oeongx@gmail.com. Thank you for your kind guidance.

an9236868 commented 2 years ago

Thanks @matthewwicker for your updates, in the analysis.py the output results are only for IBP lower bound in the chart, is that right? if yes then where can I find the LBP lower bound that you plot in the chart as well? Thanks.