search

fra31 / sparse-rs

Sparse-RS: a versatile framework for query-efficient sparse black-box adversarial attacks
https://arxiv.org/abs/2006.12834
MIT License
44 stars 6 forks source link

No dataset setup information #4

Closed loris2222 closed 3 years ago

loris2222 commented 3 years ago

Hi, I was not able to find information on how to setup imagenet to replicate results. I have tried placing the validation images in a folder and then divide them in subfolders following the rule dataset_root/class_index/image_name.JPEG as from the code I saw that you are using torchvision.datasets.ImageFolder, but it does not work. In this setup, I am getting: image

fra31 commented 3 years ago

Hi,

as you mentioned, I set the data path here to point to the folder with subfolders for each class (an example path to images is /home/scratch/datasets/imagenet/val/n01735189/ILSVRC2012_val_00016144.JPEG). From the screenshot is seems that the initial clean accuracy gets computed but is very low (it seems to be 1 image correctly classified out of 500). Could you maybe check things like the shape of the loaded batch? Are you using the pre-trained PyTorch models?

loris2222 commented 3 years ago

I found the problem. There are two different ordering of classes for imagenet. One uses the original ILSVRC class ordering, while the one used in torch's pretrained models is a new one developed by Caffe. This post explains it https://stackoverflow.com/questions/42537483/ilsvrc2012-validation-ground-truth-labels-index-not-matching-pretrained-networks I am unsure why they had to make it more complicated than it should, but anyways, thanks for your prompt reply!