reproduced results

[psteinb]

thanks for being so open and sharing all the details of your analysis (code, weights). I believe this is a super important paper.

During the reproduction, I found a couple of things that may alter some of the arguments brought forward. The slidedeck is available here: https://psteinb.github.io/adversarial-medical-imaging-imb/index.html If possible, I'd love to hear your feedback on this in case I misinterpreted your article. Please use the repo of the slides or this issue for this.

Thanks!

[sgfin]

Thanks for reproducing the work!

Just wanted to clarify what you mean by the things that impact arguments. I see two possibilities in the slides:

• Potential typo in the paper w.r.t. Inception vs Resnet
• Observation that “black box” model is better described as a “gray box”

Anything else? I’ll respond to these below, but always want to be sure I didn’t miss anything.

On the first point, I’ll be sure to check into this and update the paper, if necessary.

On this second point, I totally agree, actually. In the original paper, I had a sentence pointing out this was in fact a “gray” box, but that I was using “Black Box” since this is the term used by the paper I was citing for the method. However, I think that ended up being cut for space on a subsequent update and the Biggio paper has done a good job laying out good terminology for black/white/gray that is more standard now.

I’ve been meaning to sit down and code up a pure black box attack and add to the paper, but haven’t found the time to do so yet given other projects. Looks like you’ve already written this code — if you aren’t planning to write up your work in its own paper, I’m more than happy to add you as an author on this one if you’d like to clean up the code for the repo and add the results to the preprint. Of course that’s only if you’re interested in doing so and it’d be helpful to you.

Finally, you may be interested in some additional thoughts on this space that I’ve written here https://sgfin.github.io/2019/03/21/FAQ-On-Adversarial-Science-Paper/ and with colleagues in a Policy Forum article here https://science.sciencemag.org/content/363/6433/1287

Best, Sam

[psteinb]

Hi, thanks for replying so quickly. I appreciate that. I am more than happy to contribute and add my thoughts to a subsequent write-up or article as co-author. I think it would be worth documenting black-box attacks on classifiers with these datasets as the latter were very well chosen and have high relevance to reality in patients, doctors and society. In this regard, the diabetic retinopathy dataset needs some more thought. As I mention in the slides, the dataset used (merged trained and test samples) is highly imbalanced. From my experiments, there appears to be evidence that this alone facilitates adversarial examples. Second of all, the resize to imagenet shapes in the preprocessing pipeline adds another advantage to the adversary as the structures to indicate diabetes appear at very small scales. These two issues allowed me to flip 3 pixels out of 50176 to max/min values in a lot of images and all of a sudden the predicted class flips as well. I think both issues can be addressed, which I am currently doing. I guess, first results may be available mid/end August.

In contrast, the Xray dataset appears very robust to simple black box attacks. But they are still possible. Depending on the environment, a black box attack requires several hundred calls to the predict method. Translated to software installed close to a xray scanners in a clinic, I feel that these attacks could be detected or prevented. Still, I believe this is an important aspect to point out.

I haven't had the time to attack the last of the 3 datasets regarding skin cancer. Given the diverse set of image modalities, I am curious what I/we? can find there.

I think, this is super important work you are doing and I would be more than honored if I could contribute. Please let me know how to proceed (maybe offline).

[sgfin]

Sounds good. Send me an email at samuel_finlayson@hms.harvard.edu and we can follow up. To be clear, I don't have a ton of bandwidth for this project anymore, but I'm happy to chat and help you if I can.