Import model for CIFAR code

[maithal]

Hi,

Thank you for providing separate code for CIFAR. However, are you using models folder similar to https://github.com/kuangliu/pytorch-cifar when you say "import models" in the initial lines? If not, can you specify what exactly is import models in this part of code?

Thanks!

[cg563]

Yes. https://github.com/cg563/simple-blackbox-attack/blob/master/run_simba_cifar.py#L10 adds the pytorch-cifar subdirectory to the system path and import models should be able to import form pytorch-cifar.

[maithal]

I'll rerun the resnet18 model and check with the checkpoint. However, though it not under the same issue, can you please let me know why my targeted attack simulation results look different from yours?

Iteration 10: queries = 14.5833, prob = 0.0000, remaining = 1.0000 Iteration 20: queries = 29.8333, prob = 0.0000, remaining = 1.0000 Iteration 30: queries = 46.5833, prob = 0.0000, remaining = 1.0000 Iteration 40: queries = 61.0833, prob = 0.0000, remaining = 1.0000 Iteration 50: queries = 74.0833, prob = 0.0000, remaining = 1.0000 Iteration 60: queries = 88.9167, prob = 0.0000, remaining = 1.0000 Iteration 70: queries = 104.4167, prob = 0.0000, remaining = 1.0000 Iteration 80: queries = 120.7500, prob = 0.0000, remaining = 1.0000 Iteration 90: queries = 137.0833, prob = 0.0000, remaining = 1.0000 Iteration 100: queries = 154.4167, prob = 0.0000, remaining = 1.0000 Iteration 110: queries = 170.0833, prob = 0.0000, remaining = 1.0000 Iteration 120: queries = 186.8333, prob = 0.0000, remaining = 1.0000 Iteration 130: queries = 201.6667, prob = 0.0000, remaining = 1.0000 Iteration 140: queries = 217.0000, prob = 0.0000, remaining = 1.0000 Iteration 150: queries = 232.9167, prob = 0.0000, remaining = 1.0000 Iteration 160: queries = 249.4167, prob = 0.0000, remaining = 1.0000 Iteration 170: queries = 266.2500, prob = 0.0000, remaining = 1.0000 Iteration 180: queries = 280.7500, prob = 0.0000, remaining = 1.0000 Iteration 190: queries = 295.3333, prob = 0.0000, remaining = 1.0000 Iteration 200: queries = 312.0833, prob = 0.0000, remaining = 1.0000 Iteration 210: queries = 329.2500, prob = 0.0000, remaining = 1.0000 Iteration 220: queries = 345.7500, prob = 0.0000, remaining = 1.0000 Iteration 230: queries = 363.6667, prob = 0.0000, remaining = 1.0000 Iteration 240: queries = 380.6667, prob = 0.0001, remaining = 1.0000 Iteration 250: queries = 398.0000, prob = 0.0001, remaining = 1.0000 Iteration 260: queries = 415.0833, prob = 0.0001, remaining = 1.0000 Iteration 270: queries = 431.9167, prob = 0.0001, remaining = 1.0000 Iteration 280: queries = 448.7500, prob = 0.0001, remaining = 1.0000 Iteration 290: queries = 466.3333, prob = 0.0001, remaining = 1.0000 Iteration 300: queries = 482.7500, prob = 0.0001, remaining = 1.0000 Iteration 310: queries = 499.9167, prob = 0.0002, remaining = 1.0000 Iteration 320: queries = 515.1667, prob = 0.0002, remaining = 1.0000 Iteration 330: queries = 531.7500, prob = 0.0002, remaining = 1.0000 Iteration 340: queries = 548.1667, prob = 0.0002, remaining = 1.0000 Iteration 350: queries = 564.8333, prob = 0.0003, remaining = 1.0000 Iteration 360: queries = 582.4167, prob = 0.0003, remaining = 1.0000 Iteration 370: queries = 599.3333, prob = 0.0003, remaining = 1.0000 Iteration 380: queries = 617.3333, prob = 0.0004, remaining = 1.0000 Iteration 390: queries = 634.1667, prob = 0.0004, remaining = 1.0000 Iteration 400: queries = 651.2500, prob = 0.0005, remaining = 1.0000 Iteration 410: queries = 667.4167, prob = 0.0006, remaining = 1.0000 Iteration 420: queries = 683.8333, prob = 0.0006, remaining = 1.0000 Iteration 430: queries = 701.0000, prob = 0.0007, remaining = 1.0000 Iteration 440: queries = 718.9167, prob = 0.0008, remaining = 1.0000 Iteration 450: queries = 736.1667, prob = 0.0008, remaining = 1.0000 Iteration 460: queries = 753.5833, prob = 0.0009, remaining = 1.0000 Iteration 470: queries = 770.5000, prob = 0.0010, remaining = 1.0000 Iteration 480: queries = 788.4167, prob = 0.0011, remaining = 1.0000 Iteration 490: queries = 805.2500, prob = 0.0012, remaining = 1.0000 Iteration 500: queries = 821.7500, prob = 0.0014, remaining = 1.0000 Iteration 510: queries = 838.0000, prob = 0.0014, remaining = 1.0000 Iteration 520: queries = 856.2500, prob = 0.0016, remaining = 1.0000 Iteration 530: queries = 874.2500, prob = 0.0016, remaining = 1.0000 Iteration 540: queries = 891.3333, prob = 0.0018, remaining = 1.0000 Iteration 550: queries = 908.0000, prob = 0.0019, remaining = 1.0000 Iteration 560: queries = 924.2500, prob = 0.0022, remaining = 1.0000 Iteration 570: queries = 942.0000, prob = 0.0024, remaining = 1.0000 Iteration 580: queries = 958.4167, prob = 0.0024, remaining = 1.0000 Iteration 590: queries = 976.1667, prob = 0.0026, remaining = 1.0000 Iteration 600: queries = 993.3333, prob = 0.0027, remaining = 1.0000 Iteration 610: queries = 1010.4167, prob = 0.0028, remaining = 1.0000 Iteration 620: queries = 1026.4166, prob = 0.0029, remaining = 1.0000 Iteration 630: queries = 1043.9166, prob = 0.0031, remaining = 1.0000 Iteration 640: queries = 1061.9166, prob = 0.0033, remaining = 1.0000 Iteration 650: queries = 1080.1666, prob = 0.0035, remaining = 1.0000 Iteration 660: queries = 1098.5000, prob = 0.0036, remaining = 1.0000 Iteration 670: queries = 1116.3334, prob = 0.0038, remaining = 1.0000 Iteration 680: queries = 1133.6666, prob = 0.0040, remaining = 1.0000 Iteration 690: queries = 1151.4166, prob = 0.0043, remaining = 1.0000 Iteration 700: queries = 1169.0000, prob = 0.0045, remaining = 1.0000 Iteration 710: queries = 1186.1666, prob = 0.0048, remaining = 1.0000 Iteration 720: queries = 1203.5000, prob = 0.0052, remaining = 1.0000 Iteration 730: queries = 1221.6666, prob = 0.0055, remaining = 1.0000 Iteration 740: queries = 1238.7500, prob = 0.0058, remaining = 1.0000 Iteration 750: queries = 1256.7500, prob = 0.0061, remaining = 1.0000 Iteration 760: queries = 1273.6666, prob = 0.0065, remaining = 1.0000 Iteration 770: queries = 1290.8334, prob = 0.0067, remaining = 1.0000 Iteration 780: queries = 1308.4166, prob = 0.0069, remaining = 1.0000 Iteration 790: queries = 1325.4166, prob = 0.0072, remaining = 1.0000 Iteration 800: queries = 1343.5000, prob = 0.0075, remaining = 1.0000 Iteration 810: queries = 1360.0834, prob = 0.0080, remaining = 1.0000 Iteration 820: queries = 1377.5834, prob = 0.0083, remaining = 1.0000 Iteration 830: queries = 1394.7500, prob = 0.0086, remaining = 1.0000 Iteration 840: queries = 1412.9166, prob = 0.0089, remaining = 1.0000 Iteration 850: queries = 1430.6666, prob = 0.0094, remaining = 1.0000 Iteration 860: queries = 1447.5834, prob = 0.0100, remaining = 1.0000 Iteration 870: queries = 1465.5834, prob = 0.0105, remaining = 1.0000 Iteration 880: queries = 1483.6666, prob = 0.0111, remaining = 1.0000 Iteration 890: queries = 1500.5834, prob = 0.0115, remaining = 1.0000 Iteration 900: queries = 1516.7500, prob = 0.0122, remaining = 1.0000 Iteration 910: queries = 1532.5000, prob = 0.0130, remaining = 1.0000 Iteration 920: queries = 1550.0000, prob = 0.0133, remaining = 1.0000 Iteration 930: queries = 1567.3334, prob = 0.0137, remaining = 1.0000 Iteration 940: queries = 1584.9166, prob = 0.0142, remaining = 1.0000 Iteration 950: queries = 1602.6666, prob = 0.0148, remaining = 1.0000 Iteration 960: queries = 1620.1666, prob = 0.0157, remaining = 1.0000 Iteration 970: queries = 1637.2500, prob = 0.0161, remaining = 1.0000 Iteration 980: queries = 1655.1666, prob = 0.0171, remaining = 1.0000 Iteration 990: queries = 1673.3334, prob = 0.0180, remaining = 1.0000 Iteration 1000: queries = 1691.1666, prob = 0.0186, remaining = 1.0000 Iteration 1010: queries = 1709.5834, prob = 0.0197, remaining = 1.0000 Iteration 1020: queries = 1727.6666, prob = 0.0206, remaining = 1.0000 Iteration 1030: queries = 1745.2500, prob = 0.0217, remaining = 1.0000 Iteration 1040: queries = 1762.7500, prob = 0.0223, remaining = 1.0000 Iteration 1050: queries = 1780.3334, prob = 0.0233, remaining = 1.0000 Iteration 1060: queries = 1798.1666, prob = 0.0244, remaining = 1.0000 Iteration 1070: queries = 1815.5834, prob = 0.0251, remaining = 1.0000 Iteration 1080: queries = 1833.8334, prob = 0.0261, remaining = 1.0000 Iteration 1090: queries = 1852.3334, prob = 0.0268, remaining = 1.0000 Iteration 1100: queries = 1870.4166, prob = 0.0278, remaining = 1.0000 Iteration 1110: queries = 1888.3334, prob = 0.0291, remaining = 1.0000 Iteration 1120: queries = 1905.9166, prob = 0.0304, remaining = 1.0000 Iteration 1130: queries = 1922.8334, prob = 0.0319, remaining = 1.0000 Iteration 1140: queries = 1941.0834, prob = 0.0328, remaining = 1.0000 Iteration 1150: queries = 1959.4166, prob = 0.0341, remaining = 1.0000 Iteration 1160: queries = 1976.3334, prob = 0.0360, remaining = 0.9167 Iteration 1170: queries = 1992.5000, prob = 0.0368, remaining = 0.9167 Iteration 1180: queries = 2008.5000, prob = 0.0374, remaining = 0.9167 Iteration 1190: queries = 2025.2500, prob = 0.0383, remaining = 0.9167 Iteration 1200: queries = 2041.1666, prob = 0.0391, remaining = 0.9167 Iteration 1210: queries = 2057.8333, prob = 0.0399, remaining = 0.9167 Iteration 1220: queries = 2073.8333, prob = 0.0407, remaining = 0.9167 Iteration 1230: queries = 2089.5833, prob = 0.0417, remaining = 0.9167 Iteration 1240: queries = 2105.1667, prob = 0.0429, remaining = 0.9167 Iteration 1250: queries = 2121.0833, prob = 0.0445, remaining = 0.9167 Iteration 1260: queries = 2137.6667, prob = 0.0459, remaining = 0.9167 Iteration 1270: queries = 2153.2500, prob = 0.0474, remaining = 0.9167 Iteration 1280: queries = 2170.4167, prob = 0.0486, remaining = 0.9167 Iteration 1290: queries = 2186.6667, prob = 0.0493, remaining = 0.9167 Iteration 1300: queries = 2203.2500, prob = 0.0502, remaining = 0.9167 Iteration 1310: queries = 2219.1667, prob = 0.0513, remaining = 0.9167 Iteration 1320: queries = 2236.2500, prob = 0.0523, remaining = 0.9167 Iteration 1330: queries = 2252.6667, prob = 0.0534, remaining = 0.9167 Iteration 1340: queries = 2268.3333, prob = 0.0546, remaining = 0.9167 Iteration 1350: queries = 2284.8333, prob = 0.0560, remaining = 0.9167 Iteration 1360: queries = 2300.0000, prob = 0.0578, remaining = 0.8333 Iteration 1370: queries = 2314.6667, prob = 0.0587, remaining = 0.8333 Iteration 1380: queries = 2329.3333, prob = 0.0598, remaining = 0.8333 Iteration 1390: queries = 2344.0833, prob = 0.0605, remaining = 0.8333 Iteration 1400: queries = 2358.8333, prob = 0.0612, remaining = 0.8333 Iteration 1410: queries = 2373.8333, prob = 0.0621, remaining = 0.8333 Iteration 1420: queries = 2388.6667, prob = 0.0637, remaining = 0.8333 Iteration 1430: queries = 2403.5833, prob = 0.0648, remaining = 0.8333 Iteration 1440: queries = 2418.5000, prob = 0.0662, remaining = 0.8333 Iteration 1450: queries = 2433.4167, prob = 0.0673, remaining = 0.8333 Iteration 1460: queries = 2449.0000, prob = 0.0680, remaining = 0.8333 Iteration 1470: queries = 2463.8333, prob = 0.0695, remaining = 0.8333 Iteration 1480: queries = 2478.8333, prob = 0.0706, remaining = 0.8333 Iteration 1490: queries = 2491.5000, prob = 0.0719, remaining = 0.7500 Iteration 1500: queries = 2505.2500, prob = 0.0727, remaining = 0.7500 Iteration 1510: queries = 2519.0833, prob = 0.0734, remaining = 0.7500 Iteration 1520: queries = 2532.6667, prob = 0.0740, remaining = 0.7500 Iteration 1530: queries = 2546.5000, prob = 0.0748, remaining = 0.7500 Iteration 1540: queries = 2559.5000, prob = 0.0756, remaining = 0.7500 Iteration 1550: queries = 2572.7500, prob = 0.0767, remaining = 0.7500 Iteration 1560: queries = 2586.4167, prob = 0.0772, remaining = 0.7500 Iteration 1570: queries = 2600.3333, prob = 0.0783, remaining = 0.7500 Iteration 1580: queries = 2613.1667, prob = 0.0792, remaining = 0.7500 Iteration 1590: queries = 2626.8333, prob = 0.0804, remaining = 0.7500 Iteration 1600: queries = 2640.5000, prob = 0.0818, remaining = 0.7500 Iteration 1610: queries = 2653.9167, prob = 0.0832, remaining = 0.7500 Iteration 1620: queries = 2667.5000, prob = 0.0840, remaining = 0.7500 Iteration 1630: queries = 2679.1667, prob = 0.0848, remaining = 0.5833 Iteration 1640: queries = 2689.8333, prob = 0.0852, remaining = 0.5833 Iteration 1650: queries = 2699.6667, prob = 0.0859, remaining = 0.5833 Iteration 1660: queries = 2710.1667, prob = 0.0864, remaining = 0.5833 Iteration 1670: queries = 2720.3333, prob = 0.0871, remaining = 0.5833 Iteration 1680: queries = 2730.1667, prob = 0.0878, remaining = 0.5833 Iteration 1690: queries = 2740.7500, prob = 0.0884, remaining = 0.5833 Iteration 1700: queries = 2751.4167, prob = 0.0892, remaining = 0.5833 Iteration 1710: queries = 2761.5000, prob = 0.0901, remaining = 0.5833 Iteration 1720: queries = 2772.4167, prob = 0.0907, remaining = 0.5833 Iteration 1730: queries = 2783.0000, prob = 0.0918, remaining = 0.5833 Iteration 1740: queries = 2793.8333, prob = 0.0925, remaining = 0.5833 Iteration 1750: queries = 2804.3333, prob = 0.0931, remaining = 0.5833 Iteration 1760: queries = 2815.0833, prob = 0.0936, remaining = 0.5833 Iteration 1770: queries = 2825.6667, prob = 0.0943, remaining = 0.5833 Iteration 1780: queries = 2836.5833, prob = 0.0956, remaining = 0.5833 Iteration 1790: queries = 2847.4167, prob = 0.0960, remaining = 0.5833 Iteration 1800: queries = 2858.1667, prob = 0.0965, remaining = 0.5833 Iteration 1810: queries = 2868.4167, prob = 0.0971, remaining = 0.5833 Iteration 1820: queries = 2879.5833, prob = 0.0975, remaining = 0.5833 Iteration 1830: queries = 2890.0833, prob = 0.0983, remaining = 0.5833 Iteration 1840: queries = 2900.8333, prob = 0.0992, remaining = 0.5833 Iteration 1850: queries = 2911.4167, prob = 0.1003, remaining = 0.5833 Iteration 1860: queries = 2921.9167, prob = 0.1021, remaining = 0.5833 Iteration 1870: queries = 2932.5833, prob = 0.1033, remaining = 0.5833 Iteration 1880: queries = 2942.8333, prob = 0.1040, remaining = 0.5000 Iteration 1890: queries = 2951.7500, prob = 0.1044, remaining = 0.5000 Iteration 1900: queries = 2960.9167, prob = 0.1052, remaining = 0.5000 Iteration 1910: queries = 2969.9167, prob = 0.1058, remaining = 0.5000 Iteration 1920: queries = 2979.2500, prob = 0.1067, remaining = 0.5000 Iteration 1930: queries = 2987.1667, prob = 0.1076, remaining = 0.5000 Iteration 1940: queries = 2996.2500, prob = 0.1084, remaining = 0.5000 Iteration 1950: queries = 3005.1667, prob = 0.1089, remaining = 0.5000 Iteration 1960: queries = 3014.7500, prob = 0.1092, remaining = 0.5000 Iteration 1970: queries = 3024.4167, prob = 0.1095, remaining = 0.5000 Iteration 1980: queries = 3033.9167, prob = 0.1100, remaining = 0.5000 Iteration 1990: queries = 3043.0000, prob = 0.1108, remaining = 0.4167 Iteration 2000: queries = 3050.5833, prob = 0.1113, remaining = 0.4167 Iteration 2010: queries = 3058.5000, prob = 0.1117, remaining = 0.4167 Iteration 2020: queries = 3066.5833, prob = 0.1120, remaining = 0.4167 Iteration 2030: queries = 3074.2500, prob = 0.1123, remaining = 0.4167 Iteration 2040: queries = 3081.9167, prob = 0.1129, remaining = 0.4167 Iteration 2050: queries = 3089.5833, prob = 0.1133, remaining = 0.4167 Iteration 2060: queries = 3097.1667, prob = 0.1135, remaining = 0.4167 Iteration 2070: queries = 3104.3333, prob = 0.1139, remaining = 0.4167 Iteration 2080: queries = 3110.4167, prob = 0.1142, remaining = 0.3333 Iteration 2090: queries = 3116.8333, prob = 0.1142, remaining = 0.3333 Iteration 2100: queries = 3122.9167, prob = 0.1144, remaining = 0.3333 Iteration 2110: queries = 3129.1667, prob = 0.1146, remaining = 0.3333 Iteration 2120: queries = 3135.3333, prob = 0.1148, remaining = 0.3333 Iteration 2130: queries = 3141.4167, prob = 0.1151, remaining = 0.3333 Iteration 2140: queries = 3147.2500, prob = 0.1154, remaining = 0.3333 Iteration 2150: queries = 3153.5000, prob = 0.1157, remaining = 0.3333 Iteration 2160: queries = 3159.3333, prob = 0.1163, remaining = 0.3333 Iteration 2170: queries = 3165.5000, prob = 0.1165, remaining = 0.3333 Iteration 2180: queries = 3171.4167, prob = 0.1167, remaining = 0.3333 Iteration 2190: queries = 3177.3333, prob = 0.1170, remaining = 0.3333 Iteration 2200: queries = 3183.9167, prob = 0.1172, remaining = 0.3333 Iteration 2210: queries = 3189.9167, prob = 0.1181, remaining = 0.3333 Iteration 2220: queries = 3196.0000, prob = 0.1183, remaining = 0.3333 Iteration 2230: queries = 3202.4167, prob = 0.1189, remaining = 0.3333 Iteration 2240: queries = 3209.0000, prob = 0.1191, remaining = 0.3333 Iteration 2250: queries = 3215.2500, prob = 0.1195, remaining = 0.3333 Iteration 2260: queries = 3221.4167, prob = 0.1203, remaining = 0.3333 Iteration 2270: queries = 3227.5833, prob = 0.1205, remaining = 0.3333 Iteration 2280: queries = 3233.5000, prob = 0.1207, remaining = 0.3333 Iteration 2290: queries = 3239.9167, prob = 0.1211, remaining = 0.3333 Iteration 2300: queries = 3246.4167, prob = 0.1213, remaining = 0.3333 Iteration 2310: queries = 3252.7500, prob = 0.1216, remaining = 0.3333 Iteration 2320: queries = 3259.0000, prob = 0.1222, remaining = 0.3333 Iteration 2330: queries = 3265.0000, prob = 0.1227, remaining = 0.3333 Iteration 2340: queries = 3271.2500, prob = 0.1236, remaining = 0.3333 Iteration 2350: queries = 3277.2500, prob = 0.1242, remaining = 0.3333 Iteration 2352: queries = 3278.3333, prob = 0.1243, remaining = 0.3333

I changed num_iter=30,000 and trying to run SimBA-DCT on ImageNET data? Again, highly appreciate your quick response!

[cg563]

Is this targeted attack on ImageNet? It looks about right to me, maybe check again after 30000 iterations?

[maithal]

Yes, it is targeted attack. My concern is when I check the mean succs value it gives me 17%.

[cg563]

The last line Iteration 2352: queries = 3278.3333, prob = 0.1243, remaining = 0.3333 indicates that the attack success rate is already 66.6% after 2352 iterations. Where did you get 17% from?