search

facebookresearch / ImageNet-Adversarial-Training

ImageNet classifier with state-of-the-art adversarial robustness
Other
676 stars 87 forks source link

low accuarcy of black box defending ImageNet2012 val dataset (with no noise) #24

Closed Huang-yihao closed 4 years ago

Huang-yihao commented 4 years ago

Hello, I have a question with the accuracy of testing ImageNet2012 val dataset.

The dataset has 50000 images. I use ResNeXt101 DenoiseAll model to test the accuracy The command used by me is python main.py --eval-directory /path/to/image/directory --prediction-file predictions.txt \ --load X101-DenoiseAll.npz -d 101 --arch ResNeXtDenoiseAll --batch 20

I have checked that the synet file is the same as used by your code "from tensorpack.dataflow.dataset import ILSVRCMeta".

I have changed the code of line 244-246 of main.py to 

count=0
acc_count=0
with open(args.prediction_file, "w") as f:
    for filename, pred_label in zip(files, results):
        f.write("{},{}\n".format(filename, pred_label))
        filename = filename.split('/')[-1]
        real_label = label_of_imagenet_name[filename]
        count+=1
        if pred_label==real_label:
            acc_count+=1
print(float(acc_count)/float(count))

for calculating the accuracy. However, the accuracy is 0.00276, which is really low.

For example, the prediction label of the image 00027103.jpeg is 968 while the ground truth label is 691. the prediction label of the image 00017311.jpeg is 352 while the ground truth label is 268.

For your convenience, here I paste more prediction labels of different images for checking my results.

ILSVRC2012_val_00027103.JPEG,968
ILSVRC2012_val_00017311.JPEG,352
ILSVRC2012_val_00048769.JPEG,950
ILSVRC2012_val_00020476.JPEG,55
ILSVRC2012_val_00034219.JPEG,868
ILSVRC2012_val_00029952.JPEG,703
ILSVRC2012_val_00014941.JPEG,858
ILSVRC2012_val_00045553.JPEG,506
ILSVRC2012_val_00022086.JPEG,868
ILSVRC2012_val_00009186.JPEG,868
ILSVRC2012_val_00047907.JPEG,769
ILSVRC2012_val_00041376.JPEG,743
ILSVRC2012_val_00009707.JPEG,525
ILSVRC2012_val_00028195.JPEG,458
ILSVRC2012_val_00009320.JPEG,572
ILSVRC2012_val_00004623.JPEG,868
ILSVRC2012_val_00042993.JPEG,458
ILSVRC2012_val_00012584.JPEG,9
ILSVRC2012_val_00006028.JPEG,700
ILSVRC2012_val_00015228.JPEG,911
ILSVRC2012_val_00047254.JPEG,624
ILSVRC2012_val_00007273.JPEG,659
ILSVRC2012_val_00019123.JPEG,868
ILSVRC2012_val_00032259.JPEG,968
ILSVRC2012_val_00017723.JPEG,868
ILSVRC2012_val_00018300.JPEG,868
ILSVRC2012_val_00009639.JPEG,458
ILSVRC2012_val_00046594.JPEG,112
ILSVRC2012_val_00020428.JPEG,61
ILSVRC2012_val_00024217.JPEG,763
ILSVRC2012_val_00034291.JPEG,61
ILSVRC2012_val_00047050.JPEG,631

Whether the code and results of my experiment are right? If there are any errors, please tell me why! Thank you very much!

ppwwyyxx commented 4 years ago

I don't know how the code is modified so I cannot comment whether your code is correct. I just ran the following command:

./main.py --eval-directory ~/data/imagenet/val/n01440764/ --prediction-file pred.txt -d 101 --arch ResNeXtDenoiseAll --batch 20 --load X101-DenoiseAll.npz

which predicts on all images with label 0. Most of the predictions in pred.txt do have label 0. So I don't think the original code has significant accuracy issue.

Huang-yihao commented 4 years ago

Thank you! However, I still confuse about the result. Could you please show me the prediction label of ILSVRC2012_val_00027103.JPEG (ground truth label is 691) and ILSVRC2012_val_00017311.JPEG (ground truth label is 268) for me to verify my result? It's really important to me.

Thanks for your help!

ppwwyyxx commented 4 years ago 
/data/imagenet/val/n02114367/ILSVRC2012_val_00013479.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00014508.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00014734.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00015030.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00015465.JPEG,227
/data/imagenet/val/n02114367/ILSVRC2012_val_00015625.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00016281.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00017311.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00017773.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00017809.JPEG,360
/data/imagenet/val/n02114367/ILSVRC2012_val_00018311.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00018872.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00022055.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00024863.JPEG,248
/data/imagenet/val/n02114367/ILSVRC2012_val_00027497.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00027951.JPEG,272
/data/imagenet/val/n02114367/ILSVRC2012_val_00028990.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00029659.JPEG,271
/data/imagenet/val/n02114367/ILSVRC2012_val_00029769.JPEG,269
/data/imagenet/val/n02114367/ILSVRC2012_val_00031414.JPEG,248
/data/imagenet/val/n02114367/ILSVRC2012_val_00032568.JPEG,269
Huang-yihao commented 4 years ago

Thank you for your patience! I have found the bug.