search

Hao840 / OFAKD

PyTorch code and checkpoints release for OFA-KD: https://arxiv.org/abs/2310.19444
81 stars 11 forks source link

Accuracy in cifar100 #19

Closed liguopeng0923 closed 5 months ago

liguopeng0923 commented 5 months ago

Hi @Hao840 ,

I reproduced the results in cifar100 with kd (teacher convnext-tiny 89.96), but they are not normal compared to your paper. For example, acc@1 is 70.78, which is lower than the reported 72.99 when the student is deit-tiny. I guess that the training settings may be incorrect when the student is vit-based. Can you give me the "args.yaml" and log file?

liguopeng0923 commented 5 months ago

Similarly, when the teacher is convnext-tiny (89.96) and the student is swin-p with KD, the acc@1 is 74 which is lower than 76.44 as you reported.

Hao840 commented 5 months ago
args (DeiT-T and Swin-P use the same configuration) ``` aa: rand-m9-mstd0.5-inc1 amp: true apex_amp: false aug_repeats: 0 aug_splits: 0 batch_size: 128 bce_loss: false bce_target_thresh: null bn_eps: null bn_momentum: null channels_last: false checkpoint_hist: 10 class_map: '' clip_grad: 5.0 clip_mode: norm color_jitter: 0.4 cooldown_epochs: 0 crop_pct: null cutmix: 1.0 cutmix_minmax: null data_dir: /cache/data/cifar/ dataset: cifar100 dataset_download: false decay_epochs: 100 decay_rate: 0.1 disguise: false dist_bn: reduce drop: 0.0 drop_block: null drop_connect: null drop_path: 0.0 epoch_repeats: 0.0 epochs: 300 eps: - 1.0 eval_metric: top1 experiment: '' fuser: '' gp: null grad_checkpointing: false hflip: 0.5 img_size: null initial_checkpoint: '' input_size: null interpolation: '' jsd_loss: false kd_criterion: kl layer_decay: null local_rank: 0 log_interval: 200 log_wandb: false lr: 0.0005 lr_cycle_decay: 0.5 lr_cycle_limit: 1 lr_cycle_mul: 1.0 lr_k_decay: 1.0 lr_noise: null lr_noise_pct: 0.67 lr_noise_std: 1.0 mean: null min_lr: 1.0e-06 mixup: 0.8 mixup_mode: batch mixup_off_epoch: 0 mixup_prob: 1.0 mixup_switch_prob: 0.5 model: swin_pico_patch4_window7_224 model_ema: true model_ema_decay: 0.99996 model_ema_force_cpu: false momentum: 0.9 native_amp: false no_aug: false no_ddp_bb: false no_prefetcher: false no_resume_opt: false num_classes: 100 ofa_name: kd opt: adamw opt_betas: - 0.9 - 0.999 opt_eps: 1.0e-08 order: - 1 output: /cache/output/ patience_epochs: 10 pin_mem: false pretrained: false ratio: - 0.75 - 1.3333333333333333 recount: 1 recovery_interval: 0 remode: pixel reprob: 0.25 resplit: false resume: '' save_images: false scale: - 0.08 - 1.0 scale_norm: false sched: cosine seed: 42 smoothing: 0.1 split_bn: false stage: - 3 start_epoch: null state_key: state_dict std: null sync_bn: false teacher: convnext_tiny teacher_ckpt: /cache/ckpt/convnext_tiny_cifar100.pth temperature: 1.0 torchscript: false train_interpolation: random train_split: train tta: 0 use_multi_epochs_loader: false val_split: validation validation_batch_size: null vflip: 0.0 warmup_epochs: 20 warmup_lr: 5.0e-07 weight_decay: 0.05 weight_feat: 0.1 weight_gt: 1.0 weight_kd: 1.0 weight_ofa: 1.0 worker_seeding: all workers: 8 ```
log for DeiT-T ``` epoch,train_loss,eval_loss,eval_top1,eval_top5 1,4.57,4.4076, 4.6800,19.0600 2,4.53,4.2916, 6.9500,23.1000 3,4.47,4.1624, 9.3100,28.7400 4,4.44,4.0592,11.5100,32.8000 5,4.42,3.9734,13.2400,35.7700 6,4.34,3.9104,13.3400,37.1300 7,4.36,3.8373,15.0300,40.6000 8,4.38,3.7988,15.5800,40.8900 9,4.36,3.7517,16.2000,41.7000 10,4.33,3.7252,16.5600,40.9300 11,4.31,3.6518,17.6800,44.5600 12,4.28,3.6446,18.0600,44.3200 13,4.33,3.5983,18.7800,46.7700 14,4.27,3.5334,21.1100,48.9700 15,4.25,3.4865,20.3500,49.1400 16,4.30,3.4695,22.1700,49.4800 17,4.21,3.4349,22.1900,50.0900 18,4.21,3.3844,22.6400,52.7800 19,4.26,3.3252,24.1400,53.7600 20,4.13,3.2910,24.1000,54.0500 21,4.16,3.2437,25.9700,55.3200 22,4.22,3.2233,26.0000,56.3600 23,4.14,3.1465,28.0900,58.9600 24,4.15,3.1380,27.9500,58.4100 25,4.22,3.1016,28.9900,59.7000 26,4.01,3.0352,30.0900,60.5700 27,4.12,3.0392,29.7400,60.4300 28,4.12,3.0069,30.1600,61.4200 29,4.01,2.9410,31.5000,62.9500 30,4.17,2.9227,32.2800,63.6900 31,4.05,2.9180,31.7800,63.3200 32,4.03,2.8340,33.7700,64.8700 33,4.13,2.8311,33.9400,65.3500 34,3.97,2.8112,34.0200,65.5600 35,4.03,2.7834,35.1200,67.0800 36,4.05,2.7739,35.1800,66.6900 37,4.03,2.7196,36.6000,67.7000 38,4.06,2.6720,37.4300,68.6500 39,3.99,2.6512,36.8000,68.8100 40,4.00,2.6243,37.4300,69.4200 41,3.87,2.6294,38.0700,70.1800 42,4.01,2.5855,38.5400,70.1000 43,3.88,2.5609,39.7300,70.9600 44,3.82,2.5219,39.7700,71.4800 45,3.93,2.4852,41.0800,73.0800 46,3.98,2.4537,41.5400,72.9600 47,3.96,2.4953,41.1600,72.8000 48,3.88,2.4951,40.8100,72.3600 49,3.96,2.3858,42.8800,74.5300 50,3.96,2.3691,43.7500,75.2300 51,3.91,2.4091,42.9100,73.8000 52,3.93,2.3434,43.9700,75.1500 53,3.87,2.3110,44.1000,74.9900 54,3.81,2.2521,46.2500,77.3200 55,3.89,2.2742,45.6600,76.3400 56,3.73,2.2339,45.9600,77.1200 57,4.01,2.2375,46.1700,77.0400 58,3.91,2.2102,47.5300,77.6700 59,3.91,2.2283,47.0000,77.8800 60,3.80,2.1601,48.6400,78.7500 61,3.99,2.1621,48.4500,78.5700 62,3.89,2.1470,48.7600,78.7100 63,3.54,2.1330,48.6700,78.5500 64,3.86,2.1128,49.3700,79.2700 65,3.89,2.1122,48.8800,79.4300 66,3.70,2.0908,50.3900,80.0200 67,3.83,2.0327,51.4400,81.1900 68,3.67,2.0433,51.4900,80.9200 69,3.76,2.0226,51.8300,81.0900 70,3.73,2.0227,51.2100,81.4400 71,3.78,1.9915,51.9600,81.1600 72,3.82,2.0080,51.9400,81.0900 73,3.84,1.9689,52.1700,81.5800 74,3.81,1.9660,52.3000,81.6400 75,3.82,1.9859,51.8800,81.7400 76,3.75,1.9144,53.6400,82.5900 77,3.78,1.8909,54.1100,82.5700 78,3.73,1.9001,54.2000,82.4500 79,3.69,1.8941,54.0500,82.5900 80,3.65,1.8905,53.8600,82.8200 81,3.77,1.8620,55.3500,83.3800 82,3.66,1.8576,55.1000,83.3600 83,3.70,1.8450,55.5700,83.7300 84,3.65,1.8184,55.8600,84.1700 85,3.57,1.8510,55.9100,83.7100 86,3.80,1.8059,55.9900,84.1200 87,3.85,1.7728,57.0700,84.7300 88,3.73,1.7912,56.7300,84.2800 89,3.67,1.7992,56.2900,84.3600 90,3.79,1.7527,57.7300,85.0300 91,3.65,1.7834,56.9900,84.4200 92,3.53,1.7421,57.6400,85.1900 93,3.56,1.7410,57.9100,85.1000 94,3.74,1.7042,58.1100,85.6000 95,3.55,1.7211,58.6300,85.9800 96,3.64,1.7300,57.7700,84.9400 97,3.39,1.6779,59.2100,86.0900 98,3.45,1.6690,59.5000,86.1700 99,3.51,1.6891,59.2800,86.3600 100,3.53,1.6786,59.3500,86.1400 101,3.51,1.6721,59.3300,86.1100 102,3.63,1.6503,60.0200,86.5400 103,3.64,1.6556,59.7500,86.4400 104,3.52,1.6243,60.2100,87.1700 105,3.51,1.6533,60.0400,86.8500 106,3.48,1.6181,60.5800,86.8500 107,3.51,1.5993,61.1500,87.1700 108,3.41,1.5995,61.6500,87.5300 109,3.40,1.5917,61.4000,87.2500 110,3.61,1.5870,61.7700,87.4700 111,3.53,1.5658,62.2300,87.9700 112,3.50,1.5727,61.6900,87.3700 113,3.65,1.5713,61.7500,87.7400 114,3.52,1.5727,61.9100,87.2400 115,3.39,1.5484,62.1800,87.9500 116,3.35,1.5888,61.9200,87.4700 117,3.48,1.5185,63.4400,88.2700 118,3.48,1.5686,61.9100,88.0600 119,3.50,1.5327,63.2000,88.3700 120,3.49,1.5318,63.3900,88.3800 121,3.51,1.5082,63.7500,88.5900 122,3.29,1.5306,63.5400,88.7800 123,3.46,1.5041,63.7500,88.8100 124,3.51,1.4959,64.0900,88.9700 125,3.51,1.5228,63.1000,88.4100 126,3.38,1.4911,64.1000,88.8100 127,3.47,1.5345,62.9700,88.3800 128,3.45,1.4894,64.6700,89.0700 129,3.51,1.4916,64.1400,88.8900 130,3.47,1.4940,64.3500,88.8600 131,3.58,1.4502,65.1300,89.2100 132,3.48,1.4876,64.2900,88.7500 133,3.61,1.4812,64.2900,88.9300 134,3.42,1.5005,63.8400,88.4900 135,3.36,1.4512,64.8600,89.4900 136,3.43,1.4834,64.9400,89.2800 137,3.22,1.4569,65.0500,89.5700 138,3.34,1.4296,65.7100,89.7400 139,3.35,1.4338,65.4400,89.8700 140,3.55,1.4325,65.7000,89.8500 141,3.48,1.4501,64.8300,89.5400 142,3.37,1.4196,65.9100,90.2200 143,3.34,1.4288,65.6400,89.9100 144,3.42,1.4217,65.8500,90.1000 145,3.41,1.4178,66.3600,90.0800 146,3.28,1.4190,66.5700,90.3000 147,3.15,1.3742,66.6700,90.5200 148,3.34,1.3774,67.0000,90.3700 149,3.26,1.3932,66.9100,90.3100 150,3.32,1.4089,66.4400,90.0800 151,3.26,1.3692,67.4600,90.5400 152,3.33,1.3742,67.4500,90.7500 153,3.47,1.3664,67.4000,90.6000 154,3.42,1.4019,66.5300,90.2500 155,3.39,1.3762,67.3700,90.5800 156,3.57,1.3898,66.8500,90.3000 157,3.39,1.3686,67.9200,90.8600 158,3.38,1.3775,67.7300,90.3700 159,3.32,1.3513,67.9600,90.9200 160,3.14,1.3602,67.5000,90.4800 161,3.16,1.3481,68.0200,90.9100 162,3.43,1.3354,67.9800,90.9800 163,3.13,1.3067,68.5000,91.4100 164,3.31,1.3346,68.1400,90.8000 165,3.33,1.3304,68.3800,90.9200 166,3.27,1.3247,68.4000,91.1700 167,3.23,1.3385,67.7000,90.9800 168,3.23,1.3138,68.6000,91.4400 169,3.42,1.3162,68.6000,91.0500 170,3.12,1.3298,68.3900,91.3300 171,3.31,1.3073,68.9800,91.3700 172,3.26,1.3159,68.9700,91.4800 173,3.36,1.3328,68.5600,91.0300 174,3.47,1.3234,68.5700,91.1600 175,3.33,1.3051,68.7100,91.3700 176,3.55,1.3236,68.9900,91.5900 177,3.15,1.3220,68.6400,91.4900 178,3.12,1.2885,68.9200,91.5800 179,3.24,1.2974,69.1000,91.8000 180,3.22,1.2955,69.2600,91.3400 181,3.24,1.2792,69.6500,91.7800 182,3.47,1.3039,69.4800,91.3600 183,3.17,1.3048,69.5300,91.4500 184,3.27,1.2890,69.6000,91.3900 185,3.09,1.2969,69.3400,91.6300 186,3.32,1.2613,69.8600,91.8300 187,3.14,1.3059,69.5000,91.4900 188,3.31,1.2808,69.7600,91.7300 189,3.24,1.2786,69.5800,91.5700 190,3.17,1.2660,70.2200,92.0400 191,3.13,1.2622,70.1600,92.0500 192,2.99,1.2549,70.4900,91.9400 193,3.15,1.2789,70.2400,92.1400 194,3.29,1.2656,70.1100,91.8700 195,3.19,1.2745,70.5600,91.6800 196,3.19,1.2725,70.2200,91.6800 197,3.30,1.2814,70.1100,91.7700 198,3.18,1.2519,70.3900,91.9700 199,3.09,1.2506,70.3000,92.0800 200,3.35,1.2652,70.5500,91.9000 201,3.20,1.2759,70.2500,91.7900 202,3.30,1.2402,70.4100,92.1600 203,3.11,1.2550,70.3000,92.0500 204,3.09,1.2461,70.3500,92.3200 205,3.09,1.2320,71.0100,92.2600 206,3.17,1.2341,70.6300,92.2000 207,3.33,1.2390,70.9600,92.2200 208,3.27,1.2748,70.3800,91.9100 209,3.39,1.2375,70.9600,92.2300 210,3.23,1.2094,71.1300,92.5600 211,3.21,1.2311,71.3600,92.4300 212,3.33,1.2361,70.9500,92.4900 213,3.23,1.2334,70.9200,92.3200 214,3.12,1.2210,71.5000,92.3900 215,3.37,1.2484,71.1300,92.4100 216,3.10,1.2235,71.3100,92.3000 217,3.27,1.2135,71.5100,92.3500 218,3.04,1.2226,71.5300,92.2200 219,3.26,1.2214,71.0300,92.2400 220,3.05,1.2283,71.2600,92.5200 221,3.29,1.2357,71.1800,92.2600 222,3.29,1.2160,71.5600,92.4800 223,3.32,1.2167,71.5400,92.3900 224,3.24,1.2236,71.1900,92.3700 225,3.22,1.2060,71.6000,92.4700 226,3.18,1.2108,71.7300,92.6000 227,2.97,1.2340,71.7800,92.4200 228,3.21,1.2104,71.9000,92.5600 229,3.15,1.2133,71.6000,92.5600 230,3.19,1.1997,71.9100,92.7500 231,3.16,1.2164,72.0100,92.4400 232,3.19,1.2064,71.7700,92.4400 233,2.91,1.2069,71.8400,92.6500 234,3.04,1.1990,72.1100,92.8000 235,3.24,1.2054,71.9200,92.6000 236,3.09,1.2067,71.9100,92.6000 237,3.31,1.2015,72.0100,92.6300 238,3.12,1.1950,72.1800,92.7400 239,3.30,1.1903,72.5000,92.6600 240,3.27,1.2107,72.0500,92.7300 241,3.11,1.1988,72.1600,92.6900 242,3.00,1.1990,72.2100,92.6900 243,2.87,1.1930,72.1600,92.7900 244,3.09,1.1995,72.1400,92.6500 245,3.19,1.2070,71.6800,92.6300 246,3.38,1.2045,71.8900,92.7900 247,3.32,1.1834,72.4200,92.8400 248,3.27,1.1857,72.4400,92.6400 249,3.20,1.1949,72.3300,92.6500 250,3.12,1.1806,72.5500,92.9200 251,3.23,1.1848,72.6400,92.7400 252,3.08,1.1868,72.3500,92.7700 253,3.01,1.1907,72.4100,92.6700 254,2.94,1.1891,72.6100,92.7900 255,3.31,1.1836,72.4800,92.7900 256,3.18,1.1886,72.5000,92.7100 257,3.02,1.1927,72.5300,92.7500 258,3.27,1.1917,72.4200,92.6800 259,2.89,1.1853,72.5800,92.7600 260,3.12,1.1774,72.6700,92.8700 261,3.04,1.1916,72.4100,92.8100 262,2.96,1.1734,72.6500,92.9000 263,3.14,1.1923,72.4000,92.7200 264,3.18,1.1682,72.6800,93.0700 265,3.07,1.1817,72.7400,92.8700 266,3.10,1.1745,72.8600,92.9100 267,3.13,1.1755,72.9200,92.9000 268,3.20,1.1786,72.6000,92.9000 269,3.29,1.1747,72.8000,92.9600 270,3.05,1.1708,72.8700,92.9600 271,3.14,1.1817,72.4900,92.8200 272,2.88,1.1854,72.5200,92.8500 273,3.00,1.1815,72.6700,92.8300 274,3.06,1.1741,72.8200,93.0400 275,3.10,1.1818,72.7000,92.8800 276,2.95,1.1800,72.8800,92.9500 277,2.89,1.1734,72.8400,93.0400 278,3.19,1.1712,72.8200,93.0200 279,3.04,1.1806,72.7400,92.9200 280,3.08,1.1739,72.7800,93.0700 281,3.18,1.1756,72.6700,92.9300 282,2.99,1.1761,72.8100,92.9500 283,3.24,1.1770,72.9900,92.9300 284,3.14,1.1687,72.7400,92.9400 285,3.03,1.1706,72.8400,93.0200 286,2.99,1.1726,72.6700,92.9600 287,2.86,1.1754,72.8500,92.9100 288,3.03,1.1731,72.7500,93.0600 289,3.01,1.1724,72.8800,92.9600 290,2.90,1.1711,72.9100,92.9400 291,3.22,1.1702,72.9900,92.9900 292,3.32,1.1727,72.9000,92.9500 293,3.31,1.1723,72.8300,93.0200 294,3.21,1.1739,72.8200,92.9500 295,3.23,1.1721,72.9300,93.0200 296,3.04,1.1708,72.9600,93.0400 297,3.32,1.1715,72.8500,93.0500 298,3.06,1.1707,72.9300,93.0700 299,3.01,1.1719,72.9600,92.9700 ```
log for swin-P ``` epoch,train_loss,eval_loss,eval_top1,eval_top5 0,4.63,4.6296,1.0800,4.8000 1,4.59,4.3800,4.5500,18.0300 2,4.52,4.2895,5.6800,20.5400 3,4.48,4.1869,7.0400,24.6000 4,4.47,4.0431,9.9100,29.9000 5,4.40,3.9751,11.1200,32.6000 6,4.40,3.8918,12.4800,35.3300 7,4.38,3.7968,14.9800,39.2600 8,4.35,3.7581,14.7900,41.3900 9,4.34,3.6962,16.1900,43.1400 10,4.34,3.6646,17.3900,44.0200 11,4.30,3.5430,19.4900,47.5500 12,4.28,3.4699,20.3800,48.4300 13,4.32,3.4597,21.6100,49.9800 14,4.27,3.3826,22.9200,51.5300 15,4.26,3.3244,24.7400,53.8700 16,4.29,3.2441,25.5900,54.8800 17,4.15,3.1730,26.8200,56.8900 18,4.09,3.1264,28.2100,58.0100 19,4.12,3.0863,28.7300,59.4700 20,4.15,3.0536,29.1700,59.8600 21,4.07,2.9678,30.8900,61.5000 22,4.07,2.9029,31.6600,63.5300 23,4.08,2.8507,32.4200,64.0300 24,4.20,2.8183,33.7700,65.8400 25,4.00,2.7404,35.9600,67.0300 26,3.96,2.6980,36.1800,68.4300 27,4.07,2.6747,36.1100,67.6700 28,4.01,2.6108,37.7400,69.4800 29,4.03,2.6021,38.6200,70.2200 30,4.03,2.5463,38.9000,70.3800 31,4.01,2.5395,39.1200,71.1900 32,3.96,2.4894,40.7100,72.5800 33,3.99,2.4372,41.5700,72.4900 34,3.87,2.3909,42.2000,73.6200 35,3.77,2.3872,42.4000,73.5000 36,3.86,2.3437,43.1500,74.7400 37,3.96,2.3222,44.1700,75.7400 38,3.96,2.2939,43.6700,75.3600 39,3.83,2.2348,44.9100,76.3600 40,3.72,2.2531,44.8800,75.9600 41,3.77,2.1963,46.4200,76.1900 42,3.75,2.1708,46.7000,77.7600 43,3.83,2.1972,46.0300,76.8800 44,3.81,2.1611,47.8100,78.2500 45,3.83,2.1274,47.7200,78.5000 46,3.81,2.1021,48.6300,78.8900 47,3.67,2.0460,49.9600,80.0700 48,3.56,2.0172,50.6800,80.0200 49,3.66,2.0118,50.9000,80.4100 50,3.73,2.0303,50.5500,80.8000 51,3.77,1.9880,50.7000,80.7000 52,3.64,1.9603,52.1200,81.4300 53,3.70,1.9948,51.4600,81.1500 54,3.72,1.9046,53.4900,82.7900 55,3.70,1.8949,53.2900,82.2600 56,3.72,1.9109,53.0600,81.9600 57,3.60,1.8629,54.6100,82.9000 58,3.76,1.8525,54.0800,83.0800 59,3.70,1.8252,54.8800,83.4900 60,3.77,1.8153,55.8900,83.9500 61,3.56,1.8483,55.1500,83.4700 62,3.73,1.8132,55.4200,83.9900 63,3.42,1.7674,56.2800,84.4100 64,3.57,1.7742,56.6100,84.7300 65,3.75,1.7719,56.8700,84.8400 66,3.67,1.7167,57.4500,85.4400 67,3.54,1.7149,57.4900,85.6800 68,3.48,1.7245,57.5300,85.0100 69,3.59,1.7209,58.0600,85.3600 70,3.71,1.6970,58.1600,85.6000 71,3.66,1.6428,59.4400,86.4300 72,3.57,1.6674,59.4300,86.3500 73,3.59,1.6366,59.7600,86.5200 74,3.17,1.5930,60.1800,86.6100 75,3.59,1.5839,60.6000,86.7100 76,3.56,1.5885,60.6900,86.8200 77,3.62,1.6304,60.0900,86.5900 78,3.54,1.6453,60.3200,86.6600 79,3.55,1.5599,61.2600,87.6700 80,3.59,1.6098,60.2200,87.0400 81,3.58,1.5675,61.6100,87.5800 82,3.63,1.5683,61.8300,87.4000 83,3.41,1.5860,61.0200,87.7300 84,3.51,1.5319,61.8900,87.8900 85,3.53,1.5122,61.9700,88.2200 86,3.63,1.5420,62.4300,87.8500 87,3.57,1.5372,62.1100,87.9700 88,3.45,1.5210,62.7000,88.1700 89,3.42,1.5301,62.1100,88.1600 90,3.66,1.4853,63.9700,88.7800 91,3.52,1.5331,62.5200,88.6800 92,3.39,1.4993,63.5500,88.8600 93,3.54,1.4766,63.5900,88.7600 94,3.37,1.4584,63.8400,89.2200 95,3.53,1.4649,64.8200,89.5600 96,3.42,1.4770,64.4700,89.2600 97,3.28,1.4752,63.9800,89.5500 98,3.40,1.4553,64.1500,89.5600 99,3.11,1.4512,63.9100,89.4200 100,3.18,1.4440,64.5300,89.2700 101,3.41,1.4515,64.4200,89.4300 102,3.20,1.3937,65.4100,89.9600 103,3.31,1.3825,66.2000,90.5900 104,3.38,1.3733,66.0700,90.1400 105,3.30,1.3901,65.4600,90.3600 106,3.39,1.3806,66.2400,90.6200 107,3.40,1.4134,65.8000,90.2000 108,3.49,1.3818,66.4100,90.0900 109,3.52,1.3813,66.4400,90.6500 110,3.38,1.3945,66.9200,90.7400 111,3.34,1.3705,66.8600,91.0300 112,3.44,1.4004,66.1100,90.7700 113,3.20,1.3482,67.5600,91.0800 114,3.27,1.3553,67.5100,90.9700 115,3.19,1.3791,66.7700,90.5600 116,3.28,1.3769,66.4400,90.5600 117,3.23,1.3510,67.5500,91.1400 118,3.18,1.3171,68.0500,91.3600 119,3.38,1.2823,68.2500,91.3000 120,3.35,1.3154,68.1200,91.3300 121,3.24,1.3276,67.7800,91.5300 122,3.28,1.2802,68.3600,91.6000 123,3.41,1.3408,68.0200,91.1100 124,3.27,1.3567,67.8800,91.1200 125,3.27,1.3064,68.7900,91.5800 126,3.23,1.3423,68.6200,91.4600 127,3.25,1.2960,68.3400,91.4600 128,3.03,1.2709,68.9300,91.8200 129,3.15,1.2579,69.3700,91.9400 130,3.33,1.2707,68.8700,91.9300 131,3.17,1.3111,69.4000,91.5700 132,3.14,1.2677,69.6700,91.9300 133,3.09,1.2827,69.3100,91.7100 134,2.95,1.2368,69.0900,91.9700 135,3.28,1.2385,69.7400,92.0600 136,3.11,1.2589,69.8400,91.9200 137,3.15,1.2714,70.2700,92.1300 138,3.24,1.3100,69.9100,91.9400 139,3.22,1.2569,69.9400,92.1700 140,3.19,1.2696,70.1200,92.2000 141,3.14,1.2620,69.8900,92.2900 142,3.39,1.2699,70.1800,92.3100 143,3.33,1.2067,71.0300,92.6600 144,3.25,1.2478,70.6800,92.2500 145,3.29,1.2398,71.1000,92.6600 146,2.95,1.2296,71.0900,92.3800 147,3.08,1.2015,71.1500,92.5500 148,3.04,1.2030,70.7200,92.4200 149,3.26,1.2122,70.9300,92.5200 150,3.17,1.2333,70.7500,92.5000 151,3.13,1.2167,71.8300,92.8000 152,3.06,1.1626,71.5300,92.9900 153,3.32,1.2587,70.8100,92.7200 154,3.03,1.1926,70.6700,92.7400 155,3.19,1.1954,71.4500,92.9400 156,3.19,1.1866,71.7800,92.7900 157,3.18,1.2333,71.7300,92.8400 158,3.26,1.2429,71.7300,92.7200 159,3.36,1.2061,71.5500,92.8200 160,3.16,1.2281,71.5200,92.5500 161,3.24,1.1747,71.5300,92.9400 162,3.32,1.1458,72.4000,93.1100 163,3.20,1.1881,72.1400,93.1000 164,3.24,1.1709,72.5400,93.0600 165,3.11,1.2004,72.2400,93.0400 166,3.22,1.2100,72.2300,93.2900 167,3.25,1.2139,72.4600,93.2500 168,3.21,1.1493,72.3500,93.4300 169,3.14,1.1706,72.5100,93.4500 170,3.07,1.1697,72.2800,93.2300 171,3.11,1.1663,72.8600,93.4900 172,3.06,1.1628,72.7600,93.4200 173,2.98,1.1312,73.2300,93.6800 174,2.83,1.1458,73.1300,93.4800 175,3.19,1.1031,73.1800,93.4000 176,3.11,1.1412,73.2800,93.5400 177,2.99,1.1690,73.3000,93.4500 178,3.09,1.1324,72.7800,93.5600 179,3.25,1.1481,72.9000,93.5700 180,3.09,1.1552,73.2300,93.6300 181,3.20,1.1198,73.3600,93.5300 182,3.18,1.1578,73.5300,93.5100 183,3.02,1.1340,73.3000,93.6100 184,3.01,1.1598,73.7200,93.3900 185,3.15,1.1420,73.5600,93.6800 186,3.02,1.1336,73.3500,93.7200 187,3.02,1.1674,73.3100,93.4700 188,3.00,1.1355,73.4300,93.5700 189,3.25,1.1510,73.6000,93.7000 190,3.11,1.1279,73.7300,93.5500 191,2.76,1.1158,73.7500,94.0000 192,3.07,1.1314,73.7100,93.9400 193,3.03,1.1511,73.5600,93.8200 194,3.22,1.1232,74.5000,93.8900 195,3.15,1.1431,74.0500,93.7200 196,3.09,1.1152,74.4600,93.8900 197,3.19,1.1183,74.1600,93.9800 198,3.02,1.0793,74.2500,94.0100 199,3.02,1.1028,74.0700,93.9800 200,3.13,1.0826,74.4600,93.8200 201,2.87,1.1014,74.5000,93.8300 202,3.17,1.1104,74.6100,94.0300 203,2.92,1.1142,74.0100,94.0400 204,2.97,1.1182,74.2100,93.9100 205,3.07,1.1281,74.5300,93.9300 206,3.08,1.1121,74.6500,93.7200 207,2.77,1.1266,74.7000,94.1200 208,3.17,1.1213,74.5200,94.0700 209,2.95,1.1246,74.5500,94.0800 210,3.27,1.1298,74.2900,93.8600 211,2.95,1.1129,74.8900,94.1700 212,3.06,1.0992,75.1100,94.2400 213,3.03,1.0907,74.7300,94.1400 214,2.87,1.0759,75.2200,94.1400 215,3.09,1.1044,74.7400,94.1400 216,3.15,1.0700,75.4700,94.3400 217,3.11,1.1017,75.3600,94.1400 218,3.24,1.0872,74.8300,93.9200 219,3.15,1.0853,75.3000,94.4600 220,3.03,1.0830,75.2100,94.2300 221,3.07,1.0915,75.3000,94.0800 222,3.12,1.0909,75.0400,94.1600 223,3.13,1.0699,75.5000,94.2800 224,2.95,1.0809,75.2200,94.3000 225,2.90,1.0815,75.0200,94.1700 226,2.75,1.0799,75.3500,94.3000 227,2.85,1.0799,75.3100,94.3700 228,3.12,1.1070,75.2900,94.2000 229,3.13,1.0830,75.6700,94.2100 230,2.99,1.0575,75.5900,94.3200 231,3.09,1.1022,75.7200,94.3100 232,2.90,1.0801,75.6200,94.2700 233,2.94,1.0742,75.5000,94.2700 234,2.99,1.0950,75.4500,94.1800 235,3.10,1.1106,75.1000,94.2100 236,3.00,1.0697,75.6100,94.3900 237,3.16,1.0962,75.4300,94.2100 238,2.76,1.0765,75.8000,94.2300 239,2.95,1.0821,75.5300,94.4300 240,2.98,1.0725,75.6700,94.1500 241,2.98,1.0686,75.8800,94.4100 242,3.00,1.0544,75.5000,94.4200 243,3.01,1.0820,75.5900,94.4300 244,3.09,1.0845,75.6800,94.4300 245,2.97,1.0858,75.7900,94.3700 246,2.95,1.0690,75.8600,94.3500 247,2.88,1.0764,75.8100,94.4300 248,3.04,1.0765,75.6900,94.2700 249,2.89,1.0778,76.0900,94.5100 250,2.78,1.0756,75.9400,94.4700 251,2.93,1.0617,76.0800,94.5400 252,3.00,1.0699,76.1100,94.5500 253,2.88,1.0652,75.9900,94.4300 254,3.07,1.0578,76.2300,94.5300 255,2.85,1.0643,75.9800,94.4500 256,3.03,1.0816,76.2400,94.4600 257,2.98,1.0610,76.0400,94.5700 258,3.06,1.0941,75.7800,94.3300 259,2.93,1.0485,76.0700,94.6100 260,2.94,1.0447,76.0500,94.5700 261,2.91,1.0653,76.0700,94.5900 262,3.00,1.0540,76.2700,94.5100 263,3.09,1.0533,76.2100,94.6400 264,2.93,1.0554,76.3300,94.6400 265,3.02,1.0641,76.1400,94.5900 266,3.11,1.0497,76.1700,94.6500 267,3.06,1.0647,76.2200,94.5400 268,2.77,1.0670,76.3600,94.4800 269,3.06,1.0674,76.2100,94.6800 270,3.03,1.0558,76.2300,94.5700 271,3.12,1.0640,76.2300,94.5100 272,3.03,1.0585,76.1300,94.4900 273,2.92,1.0572,76.3100,94.5200 274,3.01,1.0531,76.2100,94.6100 275,3.20,1.0648,76.4400,94.4400 276,3.00,1.0584,76.2400,94.4600 277,2.90,1.0565,76.2400,94.5300 278,2.91,1.0593,76.1200,94.4300 279,3.19,1.0608,76.2600,94.4100 280,2.96,1.0590,76.2700,94.5100 281,2.94,1.0607,76.2800,94.4500 282,2.80,1.0598,76.2000,94.5500 283,2.86,1.0611,76.2800,94.4800 284,3.02,1.0616,76.2700,94.4900 285,3.24,1.0637,76.3100,94.5900 286,2.87,1.0574,76.1500,94.5900 287,2.97,1.0621,76.1900,94.5000 288,3.00,1.0625,76.2400,94.5300 289,3.28,1.0621,76.1600,94.5300 290,2.83,1.0619,76.2300,94.5600 291,3.23,1.0595,76.2200,94.5500 292,2.99,1.0598,76.1800,94.5300 293,2.74,1.0595,76.1800,94.5000 294,3.04,1.0583,76.1700,94.5400 295,3.00,1.0607,76.2000,94.5100 296,2.82,1.0585,76.2400,94.5200 297,2.85,1.0582,76.1900,94.5400 298,3.02,1.0577,76.2500,94.5100 299,3.09,1.0581,76.2000,94.5800 ```
liguopeng0923 commented 5 months ago

Thanks you very much !!

liguopeng0923 commented 5 months ago

Hi @Hao840 ,

I aligned the training settings and GPUs with you, but it is still not normal (DeiT-T). For example, the training loss is 6.x, but yours is 4.x at the beginning stages. Could you please check it again?

liguopeng0923 commented 5 months ago

When the teacher is convnext-tiny and the student is DeiT-T, my kd result is 71.74 but yours is 72.99 in cifar100.

liguopeng0923 commented 5 months ago

I think the main reason is the lr and batch size. After I use batch size 64, I get the normal result. Actually, this setting is the same as DEIT.