search

ShunLu91 / PA-DA

[CVPR '23] PA&DA: Jointly Sampling PAth and DAta for Consistent NAS
MIT License
34 stars 4 forks source link

nasbench201_dict.npy文件询问。 #5

Closed libingDY closed 1 month ago

libingDY commented 1 year ago

您好，请问这个nasbench201_dict.npy文件是什么作用，如果我用您的方法在新的数据集上跑，这个文件需要修改什么地方。麻烦您了。

ShunLu91 commented 1 year ago

您好，由于NAS-Bench-201官方提供的文件读取很慢，这个文件相当于把官方文件信息（主要包括搜索空间的模型结构和对应的精度，从而易于计算排序一致性）提取出来了，读取速度非常快，方便计算排序一致性。如果应用我们方法到新数据集，应该需要根据新数据集提取一个新的类似文件。

libingDY commented 1 year ago

感谢您的回复，那我应该怎么得到一个相似的文件呀？

ShunLu91 commented 1 year ago

需要收集你自己定义的搜索空间中的模型结构在新数据集的精度和结构信息，存成字典格式，最后保存为npy文件即可

libingDY commented 1 year ago

好的，麻烦您了

libingDY commented 1 year ago

10/24 11:14:54 AM model_id: 0 gene: [4, 2, 1, 2, 1, 1] loss: 0.771 top1: 73.853 target: 85.860 10/24 11:14:54 AM Evaluated: 00001 Waiting: 15624 Current Kendall's Tau: nan 10/24 11:15:43 AM model_id: 100 gene: [3, 3, 0, 3, 4, 1] loss: 0.770 top1: 77.064 target: 92.130 10/24 11:15:43 AM Evaluated: 00101 Waiting: 15524 Current Kendall's Tau: 0.11989 10/24 11:16:31 AM model_id: 200 gene: [4, 2, 0, 4, 0, 4] loss: 0.704 top1: 76.606 target: 86.970 10/24 11:16:31 AM Evaluated: 00201 Waiting: 15424 Current Kendall's Tau: 0.15659 10/24 11:17:18 AM model_id: 300 gene: [1, 2, 0, 4, 0, 2] loss: 0.845 top1: 70.642 target: 89.470 10/24 11:17:18 AM Evaluated: 00301 Waiting: 15324 Current Kendall's Tau: 0.14902 10/24 11:18:04 AM model_id: 400 gene: [1, 4, 0, 3, 0, 3] loss: 0.656 top1: 76.147 target: 92.230 10/24 11:18:04 AM Evaluated: 00401 Waiting: 15224 Current Kendall's Tau: 0.13731 10/24 11:18:51 AM model_id: 500 gene: [0, 0, 4, 3, 3, 2] loss: 0.688 top1: 78.899 target: 92.030 10/24 11:18:51 AM Evaluated: 00501 Waiting: 15124 Current Kendall's Tau: 0.12187 10/24 11:19:37 AM model_id: 600 gene: [4, 4, 2, 3, 3, 3] loss: 0.765 top1: 68.807 target: 91.860 10/24 11:19:37 AM Evaluated: 00601 Waiting: 15024 Current Kendall's Tau: 0.11459 10/24 11:20:25 AM model_id: 700 gene: [1, 0, 1, 1, 3, 0] loss: 0.677 top1: 73.853 target: 91.940 10/24 11:20:25 AM Evaluated: 00701 Waiting: 14924 Current Kendall's Tau: 0.10816 10/24 11:21:13 AM model_id: 800 gene: [4, 0, 2, 3, 2, 1] loss: 0.882 top1: 61.927 target: 91.630 10/24 11:21:13 AM Evaluated: 00801 Waiting: 14824 Current Kendall's Tau: 0.10469 10/24 11:21:59 AM model_id: 900 gene: [1, 0, 2, 1, 4, 2] loss: 0.693 top1: 72.477 target: 87.860 10/24 11:21:59 AM Evaluated: 00901 Waiting: 14724 Current Kendall's Tau: 0.10771 10/24 11:22:46 AM model_id: 1000 gene: [1, 3, 2, 0, 1, 3] loss: 0.721 top1: 74.312 target: 93.550 10/24 11:22:46 AM Evaluated: 01001 Waiting: 14624 Current Kendall's Tau: 0.11871 10/24 11:23:32 AM model_id: 1100 gene: [1, 2, 2, 0, 2, 4] loss: 1.434 top1: 53.211 target: 89.940 10/24 11:23:32 AM Evaluated: 01101 Waiting: 14524 Current Kendall's Tau: 0.10808 10/24 11:24:19 AM model_id: 1200 gene: [1, 2, 3, 4, 1, 4] loss: 0.733 top1: 73.394 target: 86.830 10/24 11:24:19 AM Evaluated: 01201 Waiting: 14424 Current Kendall's Tau: 0.09972 10/24 11:25:05 AM model_id: 1300 gene: [3, 0, 1, 4, 3, 3] loss: 0.671 top1: 77.982 target: 92.570 10/24 11:25:05 AM Evaluated: 01301 Waiting: 14324 Current Kendall's Tau: 0.10020 10/24 11:25:52 AM model_id: 1400 gene: [2, 2, 3, 2, 1, 4] loss: 1.060 top1: 66.972 target: 91.630 10/24 11:25:52 AM Evaluated: 01401 Waiting: 14224 Current Kendall's Tau: 0.09779 10/24 11:26:38 AM model_id: 1500 gene: [4, 3, 3, 0, 0, 2] loss: 1.133 top1: 52.294 target: 91.110 10/24 11:26:38 AM Evaluated: 01501 Waiting: 14124 Current Kendall's Tau: 0.09112 10/24 11:27:25 AM model_id: 1600 gene: [4, 1, 4, 4, 2, 0] loss: 0.771 top1: 72.018 target: 81.800 10/24 11:27:25 AM Evaluated: 01601 Waiting: 14024 Current Kendall's Tau: 0.09653 10/24 11:28:12 AM model_id: 1700 gene: [0, 4, 0, 4, 2, 3] loss: 0.718 top1: 72.018 target: 84.780 10/24 11:28:12 AM Evaluated: 01701 Waiting: 13924 Current Kendall's Tau: 0.09106 10/24 11:28:59 AM model_id: 1800 gene: [1, 1, 1, 3, 1, 4] loss: 0.713 top1: 72.018 target: 79.830 10/24 11:28:59 AM Evaluated: 01801 Waiting: 13824 Current Kendall's Tau: 0.09038 10/24 11:29:47 AM model_id: 1900 gene: [2, 4, 3, 0, 1, 4] loss: 0.785 top1: 72.477 target: 89.190 10/24 11:29:47 AM Evaluated: 01901 Waiting: 13724 Current Kendall's Tau: 0.08981 10/24 11:30:34 AM model_id: 2000 gene: [3, 3, 3, 3, 4, 0] loss: 0.666 top1: 74.771 target: 91.860 10/24 11:30:34 AM Evaluated: 02001 Waiting: 13624 Current Kendall's Tau: 0.09015 10/24 11:31:20 AM model_id: 2100 gene: [4, 3, 2, 4, 2, 2] loss: 0.797 top1: 69.725 target: 92.020 10/24 11:31:20 AM Evaluated: 02101 Waiting: 13524 Current Kendall's Tau: 0.08867 10/24 11:32:08 AM model_id: 2200 gene: [1, 3, 3, 1, 2, 0] loss: 0.705 top1: 72.936 target: 89.480 10/24 11:32:08 AM Evaluated: 02201 Waiting: 13424 Current Kendall's Tau: 0.08638 10/24 11:32:55 AM model_id: 2300 gene: [1, 0, 4, 3, 0, 1] loss: 0.654 top1: 77.982 target: 91.140 10/24 11:32:55 AM Evaluated: 02301 Waiting: 13324 Current Kendall's Tau: 0.07949 10/24 11:33:42 AM model_id: 2400 gene: [2, 1, 0, 1, 2, 2] loss: 0.725 top1: 68.807 target: 90.410 10/24 11:33:42 AM Evaluated: 02401 Waiting: 13224 Current Kendall's Tau: 0.07608 10/24 11:34:29 AM model_id: 2500 gene: [3, 2, 4, 3, 3, 4] loss: 0.815 top1: 70.183 target: 92.790 10/24 11:34:29 AM Evaluated: 02501 Waiting: 13124 Current Kendall's Tau: 0.07597 10/24 11:35:16 AM model_id: 2600 gene: [0, 1, 4, 3, 4, 3] loss: 0.648 top1: 77.064 target: 92.020 10/24 11:35:16 AM Evaluated: 02601 Waiting: 13024 Current Kendall's Tau: 0.07965 10/24 11:36:06 AM model_id: 2700 gene: [2, 4, 3, 3, 1, 3] loss: 0.745 top1: 73.853 target: 93.060 10/24 11:36:06 AM Evaluated: 02701 Waiting: 12924 Current Kendall's Tau: 0.07911 10/24 11:36:54 AM model_id: 2800 gene: [3, 4, 3, 1, 4, 4] loss: 0.724 top1: 73.853 target: 89.390 10/24 11:36:54 AM Evaluated: 02801 Waiting: 12824 Current Kendall's Tau: 0.07573 10/24 11:37:41 AM model_id: 2900 gene: [3, 1, 0, 2, 2, 1] loss: 0.815 top1: 67.890 target: 93.260 10/24 11:37:41 AM Evaluated: 02901 Waiting: 12724 Current Kendall's Tau: 0.07308 10/24 11:38:26 AM model_id: 3000 gene: [2, 1, 1, 0, 4, 0] loss: 0.850 top1: 69.725 target: 86.610 10/24 11:38:26 AM Evaluated: 03001 Waiting: 12624 Current Kendall's Tau: 0.07078 10/24 11:39:13 AM model_id: 3100 gene: [1, 0, 4, 0, 4, 4] loss: 0.750 top1: 73.394 target: 70.050 10/24 11:39:13 AM Evaluated: 03101 Waiting: 12524 Current Kendall's Tau: 0.07153 10/24 11:40:00 AM model_id: 3200 gene: [4, 3, 2, 3, 0, 4] loss: 0.753 top1: 75.229 target: 91.430 10/24 11:40:00 AM Evaluated: 03201 Waiting: 12424 Current Kendall's Tau: 0.07038 10/24 11:40:47 AM model_id: 3300 gene: [1, 3, 2, 2, 2, 3] loss: 0.824 top1: 71.560 target: 93.290 10/24 11:40:47 AM Evaluated: 03301 Waiting: 12324 Current Kendall's Tau: 0.07286 10/24 11:41:32 AM model_id: 3400 gene: [1, 2, 4, 2, 4, 2] loss: 0.749 top1: 69.725 target: 89.310 10/24 11:41:32 AM Evaluated: 03401 Waiting: 12224 Current Kendall's Tau: 0.07427 10/24 11:42:19 AM model_id: 3500 gene: [3, 2, 2, 0, 0, 2] loss: 1.166 top1: 56.881 target: 91.060 10/24 11:42:19 AM Evaluated: 03501 Waiting: 12124 Current Kendall's Tau: 0.07441 10/24 11:43:06 AM model_id: 3600 gene: [2, 3, 2, 4, 3, 1] loss: 0.776 top1: 73.853 target: 92.270 10/24 11:43:06 AM Evaluated: 03601 Waiting: 12024 Current Kendall's Tau: 0.07166 10/24 11:43:53 AM model_id: 3700 gene: [4, 4, 1, 3, 2, 0] loss: 0.733 top1: 74.312 target: 92.150 10/24 11:43:53 AM Evaluated: 03701 Waiting: 11924 Current Kendall's Tau: 0.07046 10/24 11:44:39 AM model_id: 3800 gene: [0, 4, 1, 1, 2, 4] loss: 0.740 top1: 72.477 target: 77.990 10/24 11:44:39 AM Evaluated: 03801 Waiting: 11824 Current Kendall's Tau: 0.06975 10/24 11:45:27 AM model_id: 3900 gene: [3, 1, 1, 2, 1, 1] loss: 0.668 top1: 74.312 target: 92.020 10/24 11:45:27 AM Evaluated: 03901 Waiting: 11724 Current Kendall's Tau: 0.06902 10/24 11:46:14 AM model_id: 4000 gene: [2, 0, 2, 3, 0, 3] loss: 0.762 top1: 68.807 target: 92.520 10/24 11:46:14 AM Evaluated: 04001 Waiting: 11624 Current Kendall's Tau: 0.06981 10/24 11:47:01 AM model_id: 4100 gene: [0, 1, 2, 1, 0, 0] loss: 0.552 top1: 79.817 target: 86.680 10/24 11:47:01 AM Evaluated: 04101 Waiting: 11524 Current Kendall's Tau: 0.07167 10/24 11:47:48 AM model_id: 4200 gene: [4, 1, 4, 0, 1, 0] loss: 0.637 top1: 74.771 target: 75.600 10/24 11:47:48 AM Evaluated: 04201 Waiting: 11424 Current Kendall's Tau: 0.07331 10/24 11:48:34 AM model_id: 4300 gene: [0, 1, 2, 2, 2, 1] loss: 0.671 top1: 76.606 target: 89.830 10/24 11:48:34 AM Evaluated: 04301 Waiting: 11324 Current Kendall's Tau: 0.07386 10/24 11:49:20 AM model_id: 4400 gene: [1, 1, 2, 3, 1, 1] loss: 0.841 top1: 63.761 target: 88.350 10/24 11:49:20 AM Evaluated: 04401 Waiting: 11224 Current Kendall's Tau: 0.07334 10/24 11:50:07 AM model_id: 4500 gene: [2, 3, 0, 4, 2, 0] loss: 0.737 top1: 70.183 target: 89.450 10/24 11:50:07 AM Evaluated: 04501 Waiting: 11124 Current Kendall's Tau: 0.07074 10/24 11:50:53 AM model_id: 4600 gene: [0, 4, 1, 4, 3, 4] loss: 0.750 top1: 73.394 target: 69.810 10/24 11:50:53 AM Evaluated: 04601 Waiting: 11024 Current Kendall's Tau: 0.06807 10/24 11:51:41 AM model_id: 4700 gene: [4, 2, 0, 4, 0, 0] loss: 0.637 top1: 74.771 target: 75.850 10/24 11:51:41 AM Evaluated: 04701 Waiting: 10924 Current Kendall's Tau: 0.06646 10/24 11:52:27 AM model_id: 4800 gene: [4, 3, 2, 1, 0, 3] loss: 0.728 top1: 74.312 target: 93.590 10/24 11:52:27 AM Evaluated: 04801 Waiting: 10824 Current Kendall's Tau: 0.06567 10/24 11:53:15 AM model_id: 4900 gene: [3, 0, 1, 3, 4, 1] loss: 0.716 top1: 74.312 target: 91.830 10/24 11:53:15 AM Evaluated: 04901 Waiting: 10724 Current Kendall's Tau: 0.06670 10/24 11:54:01 AM model_id: 5000 gene: [3, 3, 0, 4, 2, 3] loss: 0.707 top1: 74.771 target: 93.440 10/24 11:54:01 AM Evaluated: 05001 Waiting: 10624 Current Kendall's Tau: 0.06783 10/24 11:54:49 AM model_id: 5100 gene: [0, 0, 3, 0, 2, 0] loss: 1.193 top1: 55.046 target: 10.000 10/24 11:54:49 AM Evaluated: 05101 Waiting: 10524 Current Kendall's Tau: 0.06829 10/24 11:55:38 AM model_id: 5200 gene: [4, 1, 4, 4, 3, 1] loss: 0.866 top1: 66.972 target: 73.030 10/24 11:55:38 AM Evaluated: 05201 Waiting: 10424 Current Kendall's Tau: 0.06708 10/24 11:56:28 AM model_id: 5300 gene: [2, 0, 4, 4, 1, 0] loss: 0.655 top1: 76.147 target: 88.530 10/24 11:56:28 AM Evaluated: 05301 Waiting: 10324 Current Kendall's Tau: 0.06786 10/24 11:57:16 AM model_id: 5400 gene: [0, 1, 3, 1, 0, 0] loss: 0.552 top1: 79.817 target: 86.630 10/24 11:57:16 AM Evaluated: 05401 Waiting: 10224 Current Kendall's Tau: 0.06756 10/24 11:58:01 AM model_id: 5500 gene: [3, 2, 2, 1, 1, 4] loss: 0.702 top1: 74.312 target: 92.830 10/24 11:58:01 AM Evaluated: 05501 Waiting: 10124 Current Kendall's Tau: 0.06917 10/24 11:58:50 AM model_id: 5600 gene: [0, 1, 1, 3, 4, 2] loss: 0.663 top1: 76.606 target: 92.370 10/24 11:58:50 AM Evaluated: 05601 Waiting: 10024 Current Kendall's Tau: 0.06749 10/24 11:59:35 AM model_id: 5700 gene: [4, 4, 0, 1, 1, 2] loss: 0.803 top1: 68.349 target: 85.750 10/24 11:59:35 AM Evaluated: 05701 Waiting: 09924 Current Kendall's Tau: 0.06851 10/24 12:00:23 PM model_id: 5800 gene: [1, 0, 0, 4, 1, 3] loss: 0.741 top1: 69.725 target: 79.630 10/24 12:00:23 PM Evaluated: 05801 Waiting: 09824 Current Kendall's Tau: 0.06763 10/24 12:01:11 PM model_id: 5900 gene: [3, 4, 0, 3, 4, 1] loss: 0.639 top1: 79.358 target: 90.960 10/24 12:01:11 PM Evaluated: 05901 Waiting: 09724 Current Kendall's Tau: 0.06756 10/24 12:01:57 PM model_id: 6000 gene: [0, 0, 3, 4, 1, 3] loss: 0.637 top1: 74.771 target: 76.090 10/24 12:01:57 PM Evaluated: 06001 Waiting: 09624 Current Kendall's Tau: 0.06866 10/24 12:02:45 PM model_id: 6100 gene: [4, 2, 4, 1, 3, 2] loss: 0.679 top1: 74.312 target: 91.560 10/24 12:02:45 PM Evaluated: 06101 Waiting: 09524 Current Kendall's Tau: 0.06820 10/24 12:03:32 PM model_id: 6200 gene: [1, 2, 3, 4, 0, 1] loss: 0.778 top1: 70.642 target: 91.440 10/24 12:03:32 PM Evaluated: 06201 Waiting: 09424 Current Kendall's Tau: 0.06709 10/24 12:04:20 PM model_id: 6300 gene: [0, 3, 2, 0, 3, 4] loss: 0.736 top1: 74.771 target: 88.420 10/24 12:04:20 PM Evaluated: 06301 Waiting: 09324 Current Kendall's Tau: 0.06646 10/24 12:05:07 PM model_id: 6400 gene: [3, 1, 0, 2, 0, 1] loss: 0.671 top1: 76.606 target: 89.640 10/24 12:05:07 PM Evaluated: 06401 Waiting: 09224 Current Kendall's Tau: 0.06703 10/24 12:05:51 PM model_id: 6500 gene: [1, 2, 2, 4, 4, 0] loss: 0.750 top1: 72.018 target: 74.220 10/24 12:05:51 PM Evaluated: 06501 Waiting: 09124 Current Kendall's Tau: 0.06805 10/24 12:06:38 PM model_id: 6600 gene: [1, 0, 3, 0, 1, 1] loss: 0.741 top1: 66.514 target: 92.030 10/24 12:06:38 PM Evaluated: 06601 Waiting: 09024 Current Kendall's Tau: 0.06920 10/24 12:07:24 PM model_id: 6700 gene: [2, 4, 1, 0, 4, 3] loss: 0.767 top1: 70.183 target: 90.350 10/24 12:07:24 PM Evaluated: 06701 Waiting: 08924 Current Kendall's Tau: 0.06890 10/24 12:08:12 PM model_id: 6800 gene: [0, 0, 2, 2, 4, 1] loss: 0.665 top1: 73.853 target: 88.940 10/24 12:08:12 PM Evaluated: 06801 Waiting: 08824 Current Kendall's Tau: 0.06778 10/24 12:08:59 PM model_id: 6900 gene: [2, 1, 0, 3, 2, 3] loss: 0.783 top1: 70.642 target: 92.320 10/24 12:08:59 PM Evaluated: 06901 Waiting: 08724 Current Kendall's Tau: 0.06747 10/24 12:09:46 PM model_id: 7000 gene: [1, 1, 1, 3, 4, 4] loss: 0.705 top1: 72.477 target: 79.550 10/24 12:09:46 PM Evaluated: 07001 Waiting: 08624 Current Kendall's Tau: 0.06756 10/24 12:10:34 PM model_id: 7100 gene: [4, 0, 0, 4, 4, 1] loss: 0.750 top1: 73.394 target: 70.190 10/24 12:10:34 PM Evaluated: 07101 Waiting: 08524 Current Kendall's Tau: 0.06873 10/24 12:11:20 PM model_id: 7200 gene: [1, 2, 1, 3, 3, 0] loss: 0.652 top1: 76.147 target: 92.170 10/24 12:11:20 PM Evaluated: 07201 Waiting: 08424 Current Kendall's Tau: 0.06810 10/24 12:12:07 PM model_id: 7300 gene: [3, 1, 4, 2, 3, 1] loss: 0.667 top1: 74.771 target: 93.440 10/24 12:12:07 PM Evaluated: 07301 Waiting: 08324 Current Kendall's Tau: 0.06647 10/24 12:12:53 PM model_id: 7400 gene: [2, 3, 4, 4, 4, 2] loss: 0.757 top1: 75.688 target: 91.470 10/24 12:12:53 PM Evaluated: 07401 Waiting: 08224 Current Kendall's Tau: 0.06666 10/24 12:13:40 PM model_id: 7500 gene: [1, 1, 3, 3, 2, 2] loss: 0.794 top1: 71.560 target: 93.070 10/24 12:13:40 PM Evaluated: 07501 Waiting: 08124 Current Kendall's Tau: 0.06644 10/24 12:14:26 PM model_id: 7600 gene: [1, 2, 3, 1, 2, 2] loss: 0.791 top1: 68.349 target: 93.320 10/24 12:14:26 PM Evaluated: 07601 Waiting: 08024 Current Kendall's Tau: 0.06667 10/24 12:15:12 PM model_id: 7700 gene: [0, 1, 3, 2, 1, 0] loss: 0.665 top1: 73.853 target: 88.690 10/24 12:15:12 PM Evaluated: 07701 Waiting: 07924 Current Kendall's Tau: 0.06557 10/24 12:16:00 PM model_id: 7800 gene: [2, 3, 1, 3, 2, 0] loss: 0.748 top1: 72.018 target: 92.370 10/24 12:16:00 PM Evaluated: 07801 Waiting: 07824 Current Kendall's Tau: 0.06616 10/24 12:16:49 PM model_id: 7900 gene: [3, 1, 0, 1, 4, 3] loss: 0.642 top1: 77.523 target: 92.160 10/24 12:16:49 PM Evaluated: 07901 Waiting: 07724 Current Kendall's Tau: 0.06485 10/24 12:17:35 PM model_id: 8000 gene: [3, 1, 0, 1, 0, 2] loss: 0.708 top1: 71.560 target: 89.520 10/24 12:17:35 PM Evaluated: 08001 Waiting: 07624 Current Kendall's Tau: 0.06502 10/24 12:18:22 PM model_id: 8100 gene: [4, 0, 0, 2, 4, 0] loss: 0.628 top1: 76.606 target: 88.510 10/24 12:18:22 PM Evaluated: 08101 Waiting: 07524 Current Kendall's Tau: 0.06385 10/24 12:19:09 PM model_id: 8200 gene: [0, 2, 0, 2, 0, 2] loss: 0.732 top1: 69.266 target: 89.090 10/24 12:19:09 PM Evaluated: 08201 Waiting: 07424 Current Kendall's Tau: 0.06404 10/24 12:19:56 PM model_id: 8300 gene: [4, 2, 4, 2, 0, 3] loss: 0.813 top1: 69.725 target: 92.250 10/24 12:19:56 PM Evaluated: 08301 Waiting: 07324 Current Kendall's Tau: 0.06481 10/24 12:20:42 PM model_id: 8400 gene: [1, 3, 4, 4, 3, 3] loss: 0.741 top1: 76.147 target: 92.040 10/24 12:20:42 PM Evaluated: 08401 Waiting: 07224 Current Kendall's Tau: 0.06497 10/24 12:21:28 PM model_id: 8500 gene: [2, 4, 0, 0, 1, 3] loss: 0.855 top1: 70.183 target: 91.820 10/24 12:21:28 PM Evaluated: 08501 Waiting: 07124 Current Kendall's Tau: 0.06453 10/24 12:22:15 PM model_id: 8600 gene: [1, 0, 2, 4, 4, 4] loss: 0.724 top1: 73.394 target: 85.610 10/24 12:22:15 PM Evaluated: 08601 Waiting: 07024 Current Kendall's Tau: 0.06462 10/24 12:23:03 PM model_id: 8700 gene: [3, 2, 4, 0, 4, 4] loss: 0.782 top1: 72.477 target: 87.910 10/24 12:23:03 PM Evaluated: 08701 Waiting: 06924 Current Kendall's Tau: 0.06384 10/24 12:23:51 PM model_id: 8800 gene: [1, 4, 2, 4, 0, 4] loss: 0.720 top1: 73.853 target: 84.530 10/24 12:23:51 PM Evaluated: 08801 Waiting: 06824 Current Kendall's Tau: 0.06370 10/24 12:24:39 PM model_id: 8900 gene: [3, 0, 1, 4, 3, 2] loss: 0.703 top1: 76.606 target: 92.890 10/24 12:24:39 PM Evaluated: 08901 Waiting: 06724 Current Kendall's Tau: 0.06400 10/24 12:25:26 PM model_id: 9000 gene: [3, 4, 4, 4, 3, 3] loss: 0.704 top1: 74.312 target: 92.620 10/24 12:25:26 PM Evaluated: 09001 Waiting: 06624 Current Kendall's Tau: 0.06485 10/24 12:26:14 PM model_id: 9100 gene: [4, 2, 3, 2, 0, 4] loss: 1.076 top1: 56.881 target: 91.220 10/24 12:26:14 PM Evaluated: 09101 Waiting: 06524 Current Kendall's Tau: 0.06557 10/24 12:27:02 PM model_id: 9200 gene: [0, 2, 2, 1, 4, 0] loss: 0.552 top1: 79.817 target: 86.460 10/24 12:27:02 PM Evaluated: 09201 Waiting: 06424 Current Kendall's Tau: 0.06610 10/24 12:27:51 PM model_id: 9300 gene: [2, 4, 4, 3, 3, 4] loss: 0.736 top1: 76.147 target: 91.230 10/24 12:27:51 PM Evaluated: 09301 Waiting: 06324 Current Kendall's Tau: 0.06545 10/24 12:28:40 PM model_id: 9400 gene: [4, 3, 1, 1, 4, 4] loss: 0.707 top1: 73.853 target: 79.030 10/24 12:28:40 PM Evaluated: 09401 Waiting: 06224 Current Kendall's Tau: 0.06509 10/24 12:29:32 PM model_id: 9500 gene: [2, 0, 0, 2, 0, 4] loss: 0.665 top1: 73.853 target: 89.060 10/24 12:29:32 PM Evaluated: 09501 Waiting: 06124 Current Kendall's Tau: 0.06589 10/24 12:30:20 PM model_id: 9600 gene: [1, 0, 3, 3, 3, 2] loss: 0.778 top1: 69.725 target: 93.090 10/24 12:30:20 PM Evaluated: 09601 Waiting: 06024 Current Kendall's Tau: 0.06740 10/24 12:31:09 PM model_id: 9700 gene: [4, 3, 3, 2, 1, 2] loss: 0.741 top1: 72.018 target: 92.210 10/24 12:31:09 PM Evaluated: 09701 Waiting: 05924 Current Kendall's Tau: 0.06747 10/24 12:31:58 PM model_id: 9800 gene: [0, 2, 0, 2, 2, 1] loss: 0.746 top1: 70.183 target: 89.130 10/24 12:31:58 PM Evaluated: 09801 Waiting: 05824 Current Kendall's Tau: 0.06820 10/24 12:32:49 PM model_id: 9900 gene: [2, 4, 0, 3, 4, 4] loss: 0.715 top1: 71.560 target: 89.960 10/24 12:32:49 PM Evaluated: 09901 Waiting: 05724 Current Kendall's Tau: 0.06889 10/24 12:33:38 PM model_id: 10000 gene: [2, 3, 3, 1, 2, 3] loss: 0.759 top1: 67.890 target: 94.120 10/24 12:33:38 PM Evaluated: 10001 Waiting: 05624 Current Kendall's Tau: 0.06973 10/24 12:34:30 PM model_id: 10100 gene: [1, 1, 0, 2, 4, 1] loss: 0.727 top1: 71.101 target: 86.920 10/24 12:34:30 PM Evaluated: 10101 Waiting: 05524 Current Kendall's Tau: 0.07001 10/24 12:35:18 PM model_id: 10200 gene: [0, 3, 1, 1, 1, 3] loss: 0.662 top1: 77.982 target: 93.520 10/24 12:35:18 PM Evaluated: 10201 Waiting: 05424 Current Kendall's Tau: 0.06846 10/24 12:36:09 PM model_id: 10300 gene: [2, 4, 3, 1, 4, 0] loss: 0.659 top1: 78.440 target: 89.660 10/24 12:36:09 PM Evaluated: 10301 Waiting: 05324 Current Kendall's Tau: 0.06860 10/24 12:37:01 PM model_id: 10400 gene: [3, 3, 3, 2, 2, 2] loss: 0.876 top1: 70.642 target: 93.500 10/24 12:37:02 PM Evaluated: 10401 Waiting: 05224 Current Kendall's Tau: 0.06986 10/24 12:37:50 PM model_id: 10500 gene: [2, 4, 4, 3, 0, 3] loss: 0.720 top1: 72.936 target: 92.580 10/24 12:37:50 PM Evaluated: 10501 Waiting: 05124 Current Kendall's Tau: 0.06998 10/24 12:38:40 PM model_id: 10600 gene: [0, 4, 1, 2, 2, 2] loss: 0.877 top1: 63.303 target: 90.030 10/24 12:38:40 PM Evaluated: 10601 Waiting: 05024 Current Kendall's Tau: 0.06977 10/24 12:39:30 PM model_id: 10700 gene: [4, 2, 3, 4, 1, 2] loss: 0.771 top1: 70.183 target: 88.780 10/24 12:39:30 PM Evaluated: 10701 Waiting: 04924 Current Kendall's Tau: 0.06965 10/24 12:40:20 PM model_id: 10800 gene: [0, 3, 3, 0, 3, 4] loss: 0.736 top1: 74.771 target: 88.450 10/24 12:40:20 PM Evaluated: 10801 Waiting: 04824 Current Kendall's Tau: 0.06893 10/24 12:41:10 PM model_id: 10900 gene: [0, 3, 4, 1, 2, 0] loss: 0.552 top1: 79.817 target: 86.680 10/24 12:41:10 PM Evaluated: 10901 Waiting: 04724 Current Kendall's Tau: 0.06936 10/24 12:41:59 PM model_id: 11000 gene: [3, 2, 0, 1, 2, 2] loss: 0.705 top1: 75.229 target: 93.480 10/24 12:41:59 PM Evaluated: 11001 Waiting: 04624 Current Kendall's Tau: 0.07032 10/24 12:42:49 PM model_id: 11100 gene: [4, 1, 2, 4, 3, 2] loss: 0.976 top1: 61.468 target: 89.940 10/24 12:42:49 PM Evaluated: 11101 Waiting: 04524 Current Kendall's Tau: 0.07065 10/24 12:43:40 PM model_id: 11200 gene: [4, 0, 0, 0, 3, 0] loss: 0.753 top1: 71.101 target: 84.890 10/24 12:43:40 PM Evaluated: 11201 Waiting: 04424 Current Kendall's Tau: 0.07086 10/24 12:44:27 PM model_id: 11300 gene: [0, 3, 3, 1, 1, 2] loss: 0.679 top1: 75.688 target: 93.360 10/24 12:44:27 PM Evaluated: 11301 Waiting: 04324 Current Kendall's Tau: 0.07175 10/24 12:45:20 PM model_id: 11400 gene: [3, 3, 2, 1, 0, 3] loss: 0.694 top1: 75.688 target: 94.050 10/24 12:45:20 PM Evaluated: 11401 Waiting: 04224 Current Kendall's Tau: 0.07191 10/24 12:46:09 PM model_id: 11500 gene: [2, 3, 3, 3, 1, 1] loss: 0.708 top1: 71.560 target: 92.810 10/24 12:46:09 PM Evaluated: 11501 Waiting: 04124 Current Kendall's Tau: 0.07218 10/24 12:46:59 PM model_id: 11600 gene: [0, 1, 2, 1, 4, 3] loss: 0.647 top1: 75.688 target: 92.000 10/24 12:46:59 PM Evaluated: 11601 Waiting: 04024 Current Kendall's Tau: 0.07240 10/24 12:47:50 PM model_id: 11700 gene: [1, 0, 2, 0, 4, 3] loss: 0.844 top1: 64.220 target: 91.220 10/24 12:47:50 PM Evaluated: 11701 Waiting: 03924 Current Kendall's Tau: 0.07233 10/24 12:48:39 PM model_id: 11800 gene: [1, 2, 4, 2, 0, 3] loss: 0.830 top1: 70.642 target: 92.080 10/24 12:48:39 PM Evaluated: 11801 Waiting: 03824 Current Kendall's Tau: 0.07190 10/24 12:49:30 PM model_id: 11900 gene: [2, 0, 3, 3, 3, 2] loss: 0.912 top1: 62.844 target: 92.890 10/24 12:49:30 PM Evaluated: 11901 Waiting: 03724 Current Kendall's Tau: 0.07207 10/24 12:50:20 PM model_id: 12000 gene: [0, 0, 4, 3, 4, 3] loss: 0.688 top1: 78.899 target: 91.890 10/24 12:50:20 PM Evaluated: 12001 Waiting: 03624 Current Kendall's Tau: 0.07165 10/24 12:51:10 PM model_id: 12100 gene: [0, 2, 4, 0, 2, 4] loss: 0.784 top1: 68.807 target: 86.590 10/24 12:51:10 PM Evaluated: 12101 Waiting: 03524 Current Kendall's Tau: 0.07180 10/24 12:52:01 PM model_id: 12200 gene: [1, 0, 2, 4, 0, 1] loss: 0.867 top1: 63.303 target: 88.640 10/24 12:52:01 PM Evaluated: 12201 Waiting: 03424 Current Kendall's Tau: 0.07177 10/24 12:52:53 PM model_id: 12300 gene: [0, 2, 0, 1, 2, 3] loss: 0.684 top1: 76.147 target: 92.700 10/24 12:52:53 PM Evaluated: 12301 Waiting: 03324 Current Kendall's Tau: 0.07196 10/24 12:53:41 PM model_id: 12400 gene: [4, 3, 1, 2, 4, 2] loss: 0.752 top1: 71.560 target: 90.490 10/24 12:53:41 PM Evaluated: 12401 Waiting: 03224 Current Kendall's Tau: 0.07221 10/24 12:54:32 PM model_id: 12500 gene: [1, 0, 2, 3, 3, 3] loss: 0.770 top1: 72.936 target: 92.670 10/24 12:54:32 PM Evaluated: 12501 Waiting: 03124 Current Kendall's Tau: 0.07160 10/24 12:55:21 PM model_id: 12600 gene: [2, 2, 2, 4, 1, 1] loss: 0.773 top1: 74.312 target: 89.860 10/24 12:55:21 PM Evaluated: 12601 Waiting: 03024 Current Kendall's Tau: 0.07209 10/24 12:56:12 PM model_id: 12700 gene: [1, 4, 2, 1, 3, 0] loss: 0.677 top1: 73.853 target: 91.890 10/24 12:56:12 PM Evaluated: 12701 Waiting: 02924 Current Kendall's Tau: 0.07212 10/24 12:57:05 PM model_id: 12800 gene: [4, 0, 0, 4, 3, 1] loss: 0.796 top1: 70.642 target: 84.970 10/24 12:57:05 PM Evaluated: 12801 Waiting: 02824 Current Kendall's Tau: 0.07285 10/24 12:57:55 PM model_id: 12900 gene: [4, 2, 3, 3, 4, 4] loss: 0.700 top1: 76.147 target: 90.380 10/24 12:57:55 PM Evaluated: 12901 Waiting: 02724 Current Kendall's Tau: 0.07308 10/24 12:58:46 PM model_id: 13000 gene: [1, 4, 2, 2, 4, 1] loss: 0.825 top1: 68.807 target: 86.220 10/24 12:58:46 PM Evaluated: 13001 Waiting: 02624 Current Kendall's Tau: 0.07323 10/24 12:59:36 PM model_id: 13100 gene: [4, 1, 3, 0, 4, 3] loss: 0.709 top1: 74.312 target: 91.170 10/24 12:59:36 PM Evaluated: 13101 Waiting: 02524 Current Kendall's Tau: 0.07285 10/24 01:00:25 PM model_id: 13200 gene: [0, 0, 1, 2, 1, 2] loss: 0.665 top1: 73.853 target: 88.810 10/24 01:00:25 PM Evaluated: 13201 Waiting: 02424 Current Kendall's Tau: 0.07336 10/24 01:01:17 PM model_id: 13300 gene: [3, 4, 0, 4, 2, 3] loss: 0.715 top1: 71.101 target: 91.990 10/24 01:01:17 PM Evaluated: 13301 Waiting: 02324 Current Kendall's Tau: 0.07359 10/24 01:02:05 PM model_id: 13400 gene: [1, 2, 1, 4, 3, 2] loss: 0.733 top1: 75.688 target: 91.560 10/24 01:02:05 PM Evaluated: 13401 Waiting: 02224 Current Kendall's Tau: 0.07320 10/24 01:02:56 PM model_id: 13500 gene: [2, 3, 0, 0, 0, 2] loss: 0.789 top1: 69.266 target: 91.390 10/24 01:02:56 PM Evaluated: 13501 Waiting: 02124 Current Kendall's Tau: 0.07300 10/24 01:03:47 PM model_id: 13600 gene: [1, 4, 2, 3, 3, 2] loss: 0.774 top1: 72.018 target: 92.230 10/24 01:03:47 PM Evaluated: 13601 Waiting: 02024 Current Kendall's Tau: 0.07292 10/24 01:04:36 PM model_id: 13700 gene: [2, 3, 2, 3, 2, 2] loss: 0.844 top1: 67.890 target: 93.250 10/24 01:04:36 PM Evaluated: 13701 Waiting: 01924 Current Kendall's Tau: 0.07308 10/24 01:05:26 PM model_id: 13800 gene: [1, 2, 3, 2, 2, 4] loss: 1.440 top1: 53.670 target: 91.240 10/24 01:05:26 PM Evaluated: 13801 Waiting: 01824 Current Kendall's Tau: 0.07321 10/24 01:06:17 PM model_id: 13900 gene: [0, 4, 1, 4, 0, 4] loss: 0.750 top1: 73.394 target: 69.490 10/24 01:06:17 PM Evaluated: 13901 Waiting: 01724 Current Kendall's Tau: 0.07326 10/24 01:07:06 PM model_id: 14000 gene: [0, 4, 1, 3, 0, 1] loss: 0.654 top1: 77.982 target: 91.330 10/24 01:07:06 PM Evaluated: 14001 Waiting: 01624 Current Kendall's Tau: 0.07306 10/24 01:07:54 PM model_id: 14100 gene: [4, 3, 1, 1, 2, 3] loss: 0.700 top1: 73.394 target: 93.180 10/24 01:07:54 PM Evaluated: 14101 Waiting: 01524 Current Kendall's Tau: 0.07288 10/24 01:08:45 PM model_id: 14200 gene: [3, 1, 1, 1, 2, 1] loss: 0.794 top1: 63.303 target: 90.080 10/24 01:08:45 PM Evaluated: 14201 Waiting: 01424 Current Kendall's Tau: 0.07223 10/24 01:09:35 PM model_id: 14300 gene: [4, 3, 2, 3, 0, 0] loss: 0.688 top1: 78.899 target: 91.910 10/24 01:09:35 PM Evaluated: 14301 Waiting: 01324 Current Kendall's Tau: 0.07276 10/24 01:10:27 PM model_id: 14400 gene: [2, 0, 4, 4, 3, 0] loss: 0.693 top1: 74.312 target: 90.490 10/24 01:10:27 PM Evaluated: 14401 Waiting: 01224 Current Kendall's Tau: 0.07291 10/24 01:11:16 PM model_id: 14500 gene: [4, 4, 4, 4, 0, 2] loss: 0.837 top1: 68.349 target: 80.240 10/24 01:11:16 PM Evaluated: 14501 Waiting: 01124 Current Kendall's Tau: 0.07300 10/24 01:12:07 PM model_id: 14600 gene: [1, 4, 4, 2, 3, 4] loss: 0.749 top1: 73.394 target: 86.430 10/24 01:12:07 PM Evaluated: 14601 Waiting: 01024 Current Kendall's Tau: 0.07232 10/24 01:12:56 PM model_id: 14700 gene: [4, 4, 1, 2, 0, 3] loss: 0.696 top1: 71.101 target: 91.460 10/24 01:12:56 PM Evaluated: 14701 Waiting: 00924 Current Kendall's Tau: 0.07293 10/24 01:13:46 PM model_id: 14800 gene: [2, 1, 3, 2, 0, 2] loss: 1.094 top1: 57.339 target: 92.370 10/24 01:13:46 PM Evaluated: 14801 Waiting: 00824 Current Kendall's Tau: 0.07303 10/24 01:14:36 PM model_id: 14900 gene: [3, 4, 3, 1, 1, 2] loss: 0.691 top1: 74.312 target: 93.010 10/24 01:14:36 PM Evaluated: 14901 Waiting: 00724 Current Kendall's Tau: 0.07289 10/24 01:15:25 PM model_id: 15000 gene: [0, 1, 0, 0, 1, 3] loss: 0.752 top1: 69.725 target: 92.370 10/24 01:15:25 PM Evaluated: 15001 Waiting: 00624 Current Kendall's Tau: 0.07309 10/24 01:16:14 PM model_id: 15100 gene: [0, 2, 0, 4, 2, 3] loss: 0.657 top1: 76.147 target: 91.430 10/24 01:16:14 PM Evaluated: 15101 Waiting: 00524 Current Kendall's Tau: 0.07383 10/24 01:17:07 PM model_id: 15200 gene: [2, 4, 0, 0, 0, 0] loss: 1.193 top1: 55.046 target: 10.000 10/24 01:17:07 PM Evaluated: 15201 Waiting: 00424 Current Kendall's Tau: 0.07425 10/24 01:17:57 PM model_id: 15300 gene: [1, 0, 2, 0, 0, 4] loss: 1.206 top1: 51.835 target: 86.940 10/24 01:17:57 PM Evaluated: 15301 Waiting: 00324 Current Kendall's Tau: 0.07460 10/24 01:18:46 PM model_id: 15400 gene: [2, 0, 2, 3, 2, 3] loss: 0.775 top1: 71.560 target: 92.710 10/24 01:18:46 PM Evaluated: 15401 Waiting: 00224 Current Kendall's Tau: 0.07413 10/24 01:19:36 PM model_id: 15500 gene: [1, 4, 1, 0, 0, 1] loss: 0.741 top1: 69.725 target: 79.560 10/24 01:19:36 PM Evaluated: 15501 Waiting: 00124 Current Kendall's Tau: 0.07454 10/24 01:20:26 PM model_id: 15600 gene: [2, 0, 3, 3, 4, 0] loss: 0.767 top1: 76.147 target: 91.860 10/24 01:20:26 PM Evaluated: 15601 Waiting: 00024 Current Kendall's Tau: 0.07383 10/24 01:20:39 PM model_id: 15624 gene: [2, 0, 2, 0, 2, 2] loss: 1.176 top1: 55.046 target: 86.820 10/24 01:20:39 PM Evaluated: 15625 Waiting: 00000 Current Kendall's Tau: 0.07398

10/24 01:20:39 PM Finish ranking and save predictions to : results/debug_baseline.npy 10/24 01:20:39 PM Final_pred: 15625 Final_target: 15625 Final_Kendall's Tau: 0.07398 10/24 01:20:39 PM Total Elapsed Time: 5h 13min 14s

你好，我直接用cifar10作为我们的标准得到了上面的结果，上面每一个model_id对应一个测试精度，我是需要挑选出最高的精度所对应的结构重新搭建一整个网络重新训练吗？

ShunLu91 commented 1 year ago

如果目标是在这个搜索空间中找出精度最高的网络，那么就是从上边找出最高精度的网络构建网络结构重新训练即可

libingDY commented 1 year ago

您好，非常感谢您的回复，然后我没有在您提供的代码里找到相关的代码？您是否方便提供下？

ShunLu91 commented 1 year ago

您好，我们主要用这个Benchmark作为排序一致性的评估，搜索到的模型可以直接从NAS-Bench-201提供的数据里查询精度，所以没有用到相关的重训练代码。如果您要对搜索到的模型进行重训练，可以参考NAS-Bench-201的官方代码：https://github.com/D-X-Y/NAS-Bench-201

libingDY commented 1 year ago

好的，十分感谢您的回复