search

evaluating-adversarial-robustness / adv-eval-paper

LaTeX source for the paper "On Evaluating Adversarial Robustness"
https://arxiv.org/abs/1902.06705
251 stars 33 forks source link

Misclassification as malware is fun #12

Open adamshostack opened 5 years ago

adamshostack commented 5 years ago

There is a claim that "causing a benign program to be misclassified as malware may be uninteresting." It's risky to eliminate threats based on the author's understanding of attacker motivations. I suggest adding a point to "Common pitfalls" where a failure to understand attacker motivations leads to the silent elimination of attacks from consideration.

In particular, if I can convince your anti-malware engine to treat a key operating system file as malware, I can use your anti-malware engine as a denial of service amplifier of unusual power; undercut confidence in anti-malware software; damage a competitor in the market, and probably other things.

(Also works for any popular software such as Office, Chrome, Firefox, Adobe Reader)

carlini commented 5 years ago

Definitely. I agree. Causing some classifier to have a high FPR can be (and has been) seriously harmful.

I don't think we meant to say that it is uninteresting, just that some class of attackers---malware developers who want to write something new---may not care about it. And so it's important to think about if you would like to be in some setting where only one type of attack makes sense.

I think it's entirely reasonable for a paper to say "Threat Model: We only intend to prevent against someone switching some file from malicious to benign" as long as it's explicit. Definitely the silent omission of attacks is something that's bad.

However, I don't (currently) see this as a failure mode in most adversarial example defense work. Usually people are pretty clear with their threat models. They may or may not be threat models we actually care about---there is a whole debate about l_p threat models currently---but the defense work itself is usually clear about that they're focusing on an l_p threat model.

What do others think?

adamshostack commented 5 years ago

Perhaps more clear language there might be "in the space of malware detection, researchers may choose to explicitly limit their threat model to the specific...misclassified as malware may be uninteresting in that project."