Open aknirala opened 6 days ago
Please follow our readme to evaluate the pre-trained model using our code: https://github.com/Verified-Intelligence/auto_LiRPA/blob/master/doc/src/examples.md#certified-adversarial-defense-on-downscaled-imagenet-and-tinyimagenet-with-loss-fusion
I tried that first. (Had to modify the code a little where model is being loaded) Still got acc as zero. Specifically:
INFO 20:19:38 [ 1:1613]: eps=0.003921568627 CE=9.1668 Err=0.9989 Loss=9.9335 Robust_CE=9.9335 Verified_Err=0.9996 Time=0.3876
I tried using wide_resnet_imagenet64_1000, but found that it is giving zero accuracy. So, wanted to double check if this usage is correct. Here I am evaluating its natural accuracy on: Imagenet64_val_npz which I downloaded form ImageNet website (I also tried certfiied accuracy, which is also zero)
Step 1 load the model:
Step 2: Load the dataset
Step 3: Evaluate
Here is top 10 output: (No correct prediction). Q: how to evaluate it?