Section 2.4 of the paper: Training and evaluating on distortions

[DeepOceanDeep]

Hi Robert,

I have a small question about training on distortions. In the paper it is saying that DNNs were directly trained on distorted images. I was wondering if you did the distortions on the fly or saved a distorted dataset like the Stylized ImageNet?

same question for the evaluation of trained networks, when I want to run evaluate.py how can I ensure that it is comparing against a noisy image? Or the evaluation/testing is with the clean ImageNet?? I inspected the evaluation.py however, did not see any line relevant to distorting images there.

That's why I am not sure if I understood this part correctly!

[rgeirhos]

Hi @DeepOceanDeep thanks for your interest. You're right, evaluation.py does not contain code regarding testing on noisy input. If you're just interested in evaluating OOD robustness, I would recommend using https://github.com/bethgelab/model-vs-human instead which contains the distortions from rgeirhos/generalisation-humans-DNNs (and more).

If you're thinking about training your own models, you could either compute distortions on the fly or save them to disk; I guess that's mainly a speed vs. memory tradeoff: computing them on the fly will be slower but doesn't require additional storage; saving them will require more memory but training should be faster afterwards. If you choose to save the images, I would make sure to save them in a non-lossy format (e.g., png instead of jpeg) otherwise the distortions might be modified through the compression.

Does this answer your questions?

[DeepOceanDeep]

Thank @rgeirhos, I'll definitely check out the link. Just one thing I'm not quite clear on: Figure 4 displays the performance of models trained on distorted data, right? If I understand correctly, each row demonstrates how the trained model from scratch performs when tested with data containing different types of noise/distortions. The last row, for instance, indicates how well the trained models handle noisy test images with uniform noise.

So, if this is the correct interpretation, I am interested to replicate this on my model. So, I load test folder of ImageNet, and then should I apply noise to test images on the fly and see how it performs?

[rgeirhos]

Each column in this figure is one model. The model is trained on the distortions that are marked in red in this column. The model is then tested against all distortions that are shown in different rows.

So yes, for testing you could apply distortions to ImageNet images. If the model is just trained on standard ImageNet (i.e., colour images), this would mean replicating column A1. Please note that the models in this figure weren't trained on all ImageNet images, only on those corresponding to the 16 entry-level categories we used in the paper.

[DeepOceanDeep]

Thank you @rgeirhos for a detailed answer! Have a great day!

[rgeirhos]

Thanks, you too!