Deep Neural Networks Do Not Recognize Negative Images

[enricoferrero]

https://arxiv.org/abs/1703.06857

Deep Neural Networks (DNNs) have achieved remarkable performance on a variety of pattern-recognition tasks, particularly visual classification problems, where new algorithms reported to achieve or even surpass the human performance. In this paper, we test the state-of-the-art DNNs with negative images and show that the accuracy drops to the level of random classification. This leads us to the conjecture that the DNNs, which are merely trained on raw data, do not recognize the semantics of the objects, but rather memorize the inputs. We suggest that negative images can be thought as "semantic adversarial examples", which we define as transformed inputs that semantically represent the same objects, but the model does not classify them correctly.

I'm considering using this in the conclusions while highlighting the importance of the human factor.

[akundaje]

Having not read the paper, i have a question that maybe you could clarify.

Do they train on these negative images? Or just test a model trained on positive images on these negative images. If the latter, of course it will fail. That would be a trivial result.

On May 6, 2017 11:44 AM, "Enrico Ferrero" notifications@github.com wrote:

https://arxiv.org/abs/1703.06857

Deep Neural Networks (DNNs) have achieved remarkable performance on a variety of pattern-recognition tasks, particularly visual classification problems, where new algorithms reported to achieve or even surpass the human performance. In this paper, we test the state-of-the-art DNNs with negative images and show that the accuracy drops to the level of random classification. This leads us to the conjecture that the DNNs, which are merely trained on raw data, do not recognize the semantics of the objects, but rather memorize the inputs. We suggest that negative images can be thought as "semantic adversarial examples", which we define as transformed inputs that semantically represent the same objects, but the model does not classify them correctly.

I'm considering using this in the conclusions while highlighting the importance of the human factor.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7EQA3PBaOW6HJmb0V5CaJPjV3tYHUks5r3L91gaJpZM4NS3a_ .

[enricoferrero]

@akundaje Indeed, they train on positive images and then test on negative ones. As you say, they report a trivial result but I think it's interesting from a more philosophical standpoint: while in many fields DNN are surpassing human-level performance, they can also do rather "silly" mistakes such as misclassifying negative images (something a toddler would probably get right). I'm briefly touching on this in the Conclusions, see #385.

[akundaje]

I'd say this is a flaw in the machine learning problem formulation itself for this application rather than an issue with deep learning.

I think pointing out the errors deep learning and many other machine learning models make on adversarial examples that are very similar to the trained data (rather than these dramatically different test examples) is a more appropriate reference for this issue. I agree with your overall statement you are trying to make in terms of failure on adversarial examples but I think this paper is a bad example of it. I can send around a few references on in a bit if that would help.

Anshul

On May 6, 2017 12:21 PM, "Enrico Ferrero" notifications@github.com wrote:

@akundaje https://github.com/akundaje Indeed, they train on positive images and then test on negative ones. As you say, they report a trivial result but I think it's interesting from a more philosophical standpoint: while in many fields DNN are surpassing human-level performance, they can also do rather "silly" mistakes such as misclassifying negative images (something a toddler would probably get right). I'm briefly touching on this in the Conclusions, see #385 https://github.com/greenelab/deep-review/pull/385.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299660630, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7EZFkSt4js_NYQGm9ljdQzdeaz0mRks5r3MgygaJpZM4NS3a_ .

[enricoferrero]

@akundaje that would be great, thanks.

[akundaje]

Here are some key references https://arxiv.org/abs/1312.6199 (I believe this was the first paper to identify the issue of misclassification of adversarial examples that appear very similar to the correctly classified examples) https://arxiv.org/abs/1412.6572 (This paper is a follow up to the first that proposes ways to generate adversarial examples and attempt to reduce the blind spots by providing such examples in training) https://arxiv.org/abs/1703.04730 (This paper is another nice strategy using influence functions for finding adversarial examples. Fig 5 shows how an imperceptible change to a training image results in the misclassification of a different test image)

On Sat, May 6, 2017 at 12:35 PM, Enrico Ferrero notifications@github.com wrote:

@akundaje https://github.com/akundaje that would be great, thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299661332, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7EXa2Dblh1YfkLinQa7uJECiWBc11ks5r3MtugaJpZM4NS3a_ .

[agitter]

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

[akundaje]

Yes agreed.

On May 6, 2017 9:37 PM, "Anthony Gitter" notifications@github.com wrote:

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun https://github.com/qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299682002, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI7EX-tHy4E6kpoIGipIh9QtkspqvyRks5r3UqbgaJpZM4NS3a_ .

[qiyanjun]

@All, sure. I have many previous text for this. I will share them to Jack and try to make coherent to the draft.

On 5/7/17 12:37 AM, Anthony Gitter wrote:

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun https://github.com/qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299682002, or mute the thread https://github.com/notifications/unsubscribe-auth/AFb2X0CefLjvj8vNWAy215YhNdO_kq5rks5r3UqcgaJpZM4NS3a_.

[agitter]

Thanks for offering. This will be a short discussion point in the conclusion, so I think we plan to use the reference but don't need much other text.

On May 7, 2017 8:09 AM, "qiyanjun" notifications@github.com wrote:

@All, sure. I have many previous text for this. I will share them to Jack and try to make coherent to the draft.

On 5/7/17 12:37 AM, Anthony Gitter wrote:

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun https://github.com/qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384# issuecomment-299682002, or mute the thread https://github.com/notifications/unsubscribe-auth/ AFb2X0CefLjvj8vNWAy215YhNdO_kq5rks5r3UqcgaJpZM4NS3a_.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705381, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXoYiWmdOuyz5AKCDxYXzr3VBAofc2kks5r3cKZgaJpZM4NS3a_ .

[qiyanjun]

@Anthony No problem. Since we talked about this issue, please also add the following reference since it is the start of "adversarial examples":

• https://arxiv.org/abs/1312.6199

On 5/7/17 9:14 AM, Anthony Gitter wrote:

Thanks for offering. This will be a short discussion point in the conclusion, so I think we plan to use the reference but don't need much other text.

On May 7, 2017 8:09 AM, "qiyanjun" notifications@github.com wrote:

@All, sure. I have many previous text for this. I will share them to Jack and try to make coherent to the draft.

On 5/7/17 12:37 AM, Anthony Gitter wrote:

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun https://github.com/qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384# issuecomment-299682002, or mute the thread https://github.com/notifications/unsubscribe-auth/ AFb2X0CefLjvj8vNWAy215YhNdO_kq5rks5r3UqcgaJpZM4NS3a_.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705381, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXoYiWmdOuyz5AKCDxYXzr3VBAofc2kks5r3cKZgaJpZM4NS3a_ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705630, or mute the thread https://github.com/notifications/unsubscribe-auth/AFb2X_VFelB67EVM6K9TTMdXOSmghvYkks5r3cOggaJpZM4NS3a_.

[qiyanjun]

Here is another classic paper about defending against adversarial examples:

• https://arxiv.org/pdf/1412.6572.pdf

EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES

At the same time, I recommend to talk about privacy-preserving deep learning, like to cite the following two papers:

• https://www.cs.cornell.edu/~shmat/shmat_ccs15.pdf

Privacy-Preserving Deep Learning

• https://arxiv.org/abs/1607.00133

Deep Learning with Differential Privacy

On 5/7/17 9:14 AM, Anthony Gitter wrote:

Thanks for offering. This will be a short discussion point in the conclusion, so I think we plan to use the reference but don't need much other text.

On May 7, 2017 8:09 AM, "qiyanjun" notifications@github.com wrote:

@All, sure. I have many previous text for this. I will share them to Jack and try to make coherent to the draft.

On 5/7/17 12:37 AM, Anthony Gitter wrote:

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun https://github.com/qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384# issuecomment-299682002, or mute the thread https://github.com/notifications/unsubscribe-auth/ AFb2X0CefLjvj8vNWAy215YhNdO_kq5rks5r3UqcgaJpZM4NS3a_.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705381, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXoYiWmdOuyz5AKCDxYXzr3VBAofc2kks5r3cKZgaJpZM4NS3a_ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705630, or mute the thread https://github.com/notifications/unsubscribe-auth/AFb2X_VFelB67EVM6K9TTMdXOSmghvYkks5r3cOggaJpZM4NS3a_.

[qiyanjun]

@Anthony No problem. Since we talked about this issue,

please also add the following reference since it is the start of "adversarial examples":

• https://arxiv.org/abs/1312.6199

Here is another classic paper about defending against adversarial examples:

• https://arxiv.org/pdf/1412.6572.pdf

EXPLAINING AND HARNESSING ADVERSARIAL EXAMPLES

At the same time, I recommend to talk about privacy-preserving deep learning, like to cite the following two papers:

• https://www.cs.cornell.edu/~shmat/shmat_ccs15.pdf

Privacy-Preserving Deep Learning

• https://arxiv.org/abs/1607.00133

Deep Learning with Differential Privacy

On 5/7/17 9:14 AM, Anthony Gitter wrote:

Thanks for offering. This will be a short discussion point in the conclusion, so I think we plan to use the reference but don't need much other text.

On May 7, 2017 8:09 AM, "qiyanjun" notifications@github.com wrote:

@All, sure. I have many previous text for this. I will share them to Jack and try to make coherent to the draft.

On 5/7/17 12:37 AM, Anthony Gitter wrote:

If we raise the issue of adversarial examples, should we also discuss strategies for defending against them? @qiyanjun https://github.com/qiyanjun posted https://arxiv.org/abs/1704.01155 (Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks) recently, and Section 2.3 discusses other defensive techniques.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384# issuecomment-299682002, or mute the thread https://github.com/notifications/unsubscribe-auth/ AFb2X0CefLjvj8vNWAy215YhNdO_kq5rks5r3UqcgaJpZM4NS3a_.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705381, or mute the thread https://github.com/notifications/unsubscribe-auth/ADXoYiWmdOuyz5AKCDxYXzr3VBAofc2kks5r3cKZgaJpZM4NS3a_ .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/greenelab/deep-review/issues/384#issuecomment-299705630, or mute the thread https://github.com/notifications/unsubscribe-auth/AFb2X_VFelB67EVM6K9TTMdXOSmghvYkks5r3cOggaJpZM4NS3a_.

[agitter]

I'm closing this because the discussion was incorporated in #385.

@qiyanjun thanks for the additional papers. We included some (maybe all?) of these.