Training speed of mixmo is very slow

[chenshen03]

I followed your instructions to train mixmo on cifar-100. However, the training is very slow compared to the baseline. Can you tell me how much time it takes to train mixmo on CIFAR-100 in your experiments?

$ python3 scripts/train.py --config_path config/cifar100/exp_cifar100_wrn2810-2_cutmixmo-p5_msdacutmix_bar4.yaml --dataplace $dataplace --saveplace $saveplace

[alexrame]

Thank you for your interest. Indeed MixMo performs best with batch repetition (first introduced in MIMO https://arxiv.org/pdf/2010.06610.pdf). This option file has batch repetition set to 4, therefore each image is duplicated 4 times in a batch: overall, each epoch is 4 times longer. Note that we compare different approaches at equal training time in Section 4.3.1 from our paper (https://arxiv.org/pdf/2103.06132.pdf) and that most concurrent approaches also lengthen training. I hope this answers your question.

[chenshen03]

Thank you very much. Becides, I ran the experiment exp_cifar100_wrn2810-2_cutmixmo-p5_msdacutmix_bar4.yaml for several days and the best accuracy in epoch 158 is currently 92.03%, which still well below the 95.77% reported in the paper. Can you share your training log? Thank you very much!

########################################
epoch                       158
general/classifier_lr       0.00125
general/l2_reg              3.795e+03
test/accuracy               82.03%
test/accuracy_0             78.45%
test/accuracy_1             79.68%
test/accuracy_mean          79.06%
test/accuracytop5           96.26%
test/auc                    0.894054
test/brier                  0.00250124
test/ce0                    0.79515
test/ce1                    0.73879
test/diversity_ratioerrors  1.0451
test/ece                    0.0234672
test/main                   2.6724
test/nll                    0.626094
train/ce0                   1.2476
train/ce1                   1.2436
train/main                  3.6286
########################################

[alexrame]

That's totally normal to have around 82% at epoch 158. Accuracies are sharply increasing at each learning rate update, occurring at epochs100, 200, and 225.
Here I provide the logs for the best performing run that reaches 85.92 at epoch 273.

epoch,test/accuracy,test/accuracytop5,test/accuracy_mean,test/accuracy_0,test/accuracy_1,train/accuracy,checkpoint_saved 1,9.12,29.13,08.745,07.98,09.51,4.31,0.0 2,14.41,38.89,12.88,11.61,14.15,8.94,0.0 3,22.47,53.19,20.795,20.15,21.44,13.14,0.0 4,30.08,62.93,28.05,25.87,30.23,16.60,0.0 5,32.88,65.62,30.535,30.46,30.61,20.15,0.0 6,36.93,72.25,35.065,35.31,34.82,22.66,0.0 7,40.23,73.99,36.46,36.25,36.67,25.23,0.0 8,43.39,76.11,37.76,40.01,35.51,27.41,0.0 9,48.08,80.32,42.775,43.02,42.53,29.27,0.0 10,50.56,81.63,46.36,45.85,46.87,31.07,0.0 11,50.11,82.21,46.095,44.2,47.99,32.32,0.0 12,51.43,83.04,47.915,48.01,47.82,33.76,0.0 13,54.25,84.54,49.935,51.5,48.37,34.48,0.0 14,51.91,83.49,47.755,48.38,47.13,36.10,0.0 15,53.29,85.46,50.115,50.48,49.75,36.75,1.0 16,54.01,84.61,50.2,49.63,50.77,37.57,0.0 17,56.67,86.04,52.26,52.88,51.64,38.27,0.0 18,60.02,88.17,56.23,54.39,58.07,38.55,0.0 19,56.17,85.12,51.53,53.79,49.27,39.47,0.0 20,59.80,86.84,55.51,54.68,56.34,39.65,0.0 21,59.96,87.79,54.535,54.8,54.27,40.29,0.0 22,58.71,86.23,54.92,53.85,55.99,40.80,0.0 23,59.92,87.97,55.815,54.91,56.72,41.30,0.0 24,61.14,88.59,57.5,57.41,57.59,41.55,0.0 25,61.60,88.22,57.095,57.09,57.1,41.62,0.0 26,62.69,89.23,58.6,58.31,58.89,42.15,0.0 27,62.95,89.73,58.555,56.36,60.75,42.34,0.0 28,59.57,88.20,56.06,55.41,56.71,42.54,0.0 29,62.29,89.05,58.57,59.04,58.1,42.93,0.0 30,63.07,89.51,58.72,58.66,58.78,43.24,1.0 31,60.87,87.71,57.31,58.11,56.51,43.56,0.0 32,61.63,88.58,56.3,54.07,58.53,43.69,0.0 33,63.81,89.75,60.41,60.81,60.01,43.84,0.0 34,63.12,89.11,58.34,58.17,58.51,43.94,0.0 35,59.36,87.13,56.31,55.63,56.99,43.73,0.0 36,63.57,89.58,58.84,57.69,59.99,44.07,0.0 37,62.90,89.27,59.145,59.6,58.69,44.41,0.0 38,62.71,89.36,57.17,59.56,54.78,44.62,0.0 39,64.47,90.19,60.32,61.34,59.3,44.68,0.0 40,65.94,90.85,61.4,61.05,61.75,45.03,0.0 41,65.18,90.25,61.01,61.59,60.43,45.01,0.0 42,63.57,89.94,59.46,60.81,58.11,45.00,0.0 43,63.28,89.62,60.365,60.83,59.9,44.95,0.0 44,62.12,87.90,57.55,58.87,56.23,44.95,0.0 45,65.06,90.75,60.695,60.25,61.14,45.30,1.0 46,64.92,90.48,59.815,58.62,61.01,45.47,0.0 47,65.55,90.78,60.375,60.0,60.75,45.32,0.0 48,63.47,89.87,58.75,57.22,60.28,45.93,0.0 49,65.52,89.60,60.77,58.25,63.29,45.74,0.0 50,64.31,90.25,60.585,61.49,59.68,45.85,0.0 51,65.29,90.28,60.59,60.02,61.16,45.69,0.0 52,64.80,90.35,61.17,61.86,60.48,45.92,0.0 53,65.67,90.53,61.335,60.55,62.12,46.08,0.0 54,67.10,91.29,61.76,61.71,61.81,46.44,0.0 55,64.39,89.20,58.44,54.73,62.15,46.29,0.0 56,66.71,91.11,62.78,61.25,64.31,46.27,0.0 57,66.45,91.44,62.445,61.48,63.41,46.40,0.0 58,67.12,91.62,62.8,62.97,62.63,46.48,0.0 59,64.23,90.77,60.3,58.66,61.94,46.81,0.0 60,67.20,91.39,62.865,63.48,62.25,46.37,1.0 61,67.76,91.96,63.825,63.22,64.43,46.84,0.0 62,66.24,91.60,62.14,61.06,63.22,46.60,0.0 63,66.46,90.63,62.955,63.17,62.74,46.47,0.0 64,66.90,90.73,62.9,63.14,62.66,46.67,0.0 65,66.80,91.07,62.975,64.3,61.65,47.34,0.0 66,68.03,91.86,63.615,65.14,62.09,46.61,0.0 67,68.01,91.83,63.955,63.9,64.01,47.01,0.0 68,66.04,90.56,61.555,61.94,61.17,47.20,0.0 69,67.59,91.01,63.335,62.71,63.96,47.09,0.0 70,66.31,90.50,62.125,61.66,62.59,47.23,0.0 71,66.92,90.88,62.605,62.62,62.59,47.14,0.0 72,64.25,90.50,60.645,59.39,61.9,47.41,0.0 73,68.19,91.65,63.84,63.36,64.32,47.01,0.0 74,67.04,91.30,62.985,62.8,63.17,47.32,0.0 75,67.46,90.59,62.07,63.72,60.42,47.46,1.0 76,66.76,91.30,62.815,64.02,61.61,47.22,0.0 77,67.92,92.29,63.555,63.1,64.01,47.06,0.0 78,66.89,91.07,62.075,62.06,62.09,47.62,0.0 79,68.39,91.31,63.635,63.76,63.51,47.65,0.0 80,66.62,91.13,61.745,62.24,61.25,47.64,0.0 81,65.99,90.09,60.955,58.77,63.14,47.72,0.0 82,67.42,91.22,61.89,62.42,61.36,47.77,0.0 83,68.13,91.94,63.915,64.06,63.77,47.84,0.0 84,66.89,91.12,62.45,60.83,64.07,47.55,0.0 85,67.56,91.15,63.235,64.93,61.54,47.84,0.0 86,66.99,91.12,62.405,62.2,62.61,47.69,0.0 87,68.81,92.12,63.775,64.77,62.78,47.84,0.0 88,64.46,89.98,60.025,61.68,58.37,47.96,0.0 89,66.35,90.81,62.27,63.25,61.29,47.78,0.0 90,65.25,90.77,60.445,57.25,63.64,48.03,1.0 91,66.90,90.87,62.23,64.34,60.12,47.82,0.0 92,65.18,90.52,61.,003.4,61.66,47.63,0.0 93,65.92,90.82,62.07,62.39,61.75,47.99,0.0 94,68.22,92.07,63.385,62.38,64.39,47.63,0.0 95,66.10,91.15,61.32,60.5,62.14,47.82,0.0 96,67.87,90.67,64.12,63.29,64.95,47.86,0.0 97,67.17,91.33,62.935,63.97,61.9,47.95,0.0 98,67.26,90.36,62.54,63.06,62.02,48.13,0.0 99,68.97,91.59,64.42,65.62,63.22,47.75,0.0 100,65.10,90.20,61.74,62.75,60.73,47.99,0.0 101,79.10,95.88,77.37,77.18,77.56,59.90,0.0 102,80.07,96.53,78.71,78.65,78.77,62.74,0.0 103,80.65,96.46,79.065,79.24,78.89,63.99,0.0 104,80.36,96.54,78.915,78.92,78.91,64.84,0.0 105,81.46,96.67,79.79,79.76,79.82,65.56,1.0 106,81.56,96.81,79.84,79.93,79.75,66.12,0.0 107,81.91,96.87,80.275,80.57,79.98,66.70,0.0 108,81.85,96.96,80.14,80.22,80.06,67.08,0.0 109,81.72,96.90,79.995,80.24,79.75,67.52,0.0 110,81.77,96.86,80.145,79.89,80.4,68.30,0.0 111,82.15,96.87,80.19,80.19,80.19,68.23,0.0 112,81.83,96.65,80.125,80.38,79.87,68.30,0.0 113,82.52,96.74,80.215,80.28,80.15,68.83,0.0 114,82.15,96.70,80.025,79.93,80.12,69.03,0.0 115,81.81,96.64,80.11,80.07,80.15,68.96,0.0 116,82.30,96.88,80.22,80.35,80.09,68.90,0.0 117,82.43,96.92,80.245,80.25,80.24,69.78,0.0 118,82.28,96.74,80.37,80.82,79.92,69.48,0.0 119,82.53,96.90,80.45,80.68,80.22,70.12,0.0 120,81.88,96.67,79.66,79.7,79.62,69.69,1.0 121,81.87,96.61,79.45,79.27,79.63,70.21,0.0 122,81.77,96.69,79.65,79.59,79.71,70.05,0.0 123,82.21,96.79,80.05,80.22,79.88,69.80,0.0 124,81.77,96.66,78.98,78.74,79.22,70.16,0.0 125,81.86,96.86,79.505,79.55,79.46,70.79,0.0 126,82.47,96.66,80.185,79.97,80.4,70.25,0.0 127,81.86,96.64,79.505,79.64,79.37,70.34,0.0 128,81.86,96.54,79.475,79.4,79.55,70.59,0.0 129,81.95,96.70,79.405,79.16,79.65,70.46,0.0 130,82.17,96.67,79.695,79.6,79.79,70.89,0.0 131,81.63,96.66,79.05,79.31,78.79,70.86,0.0 132,81.92,96.70,78.89,78.99,78.79,71.00,0.0 133,82.31,96.56,79.725,79.69,79.76,70.70,0.0 134,82.05,96.47,79.51,79.38,79.64,70.63,0.0 135,81.57,96.39,79.23,79.13,79.33,71.36,1.0 136,82.13,96.77,79.425,79.7,79.15,71.25,0.0 137,82.48,96.62,79.685,79.51,79.86,71.21,0.0 138,82.16,96.73,79.57,80.11,79.03,71.26,0.0 139,81.66,96.24,79.145,79.18,79.11,71.55,0.0 140,82.66,96.79,80.075,80.39,79.76,71.48,0.0 141,81.95,96.40,79.63,79.27,79.99,71.43,0.0 142,82.46,96.63,79.73,79.62,79.84,71.43,0.0 143,82.03,96.35,79.08,79.2,78.96,71.58,0.0 144,81.90,96.72,79.265,79.59,78.94,71.70,0.0 145,81.24,96.44,78.32,78.97,77.67,71.83,0.0 146,81.65,96.20,78.85,78.74,78.96,71.74,0.0 147,82.18,96.61,79.46,79.52,79.4,71.76,0.0 148,81.85,96.52,78.97,79.22,78.72,71.78,0.0 149,82.49,96.58,79.605,79.77,79.44,71.87,0.0 150,82.17,96.55,79.58,79.67,79.49,71.74,1.0 151,82.69,96.58,80.02,80.05,79.99,71.94,0.0 152,81.93,96.34,79.11,79.1,79.12,72.43,0.0 153,82.00,96.42,78.96,78.61,79.31,72.34,0.0 154,82.15,96.67,79.64,79.29,79.99,72.38,0.0 155,81.31,96.62,78.72,78.84,78.6,72.67,0.0 156,81.94,96.38,79.21,79.31,79.11,71.97,0.0 157,81.84,96.47,79.11,78.79,79.43,72.51,0.0 158,82.10,96.53,79.455,79.74,79.17,72.15,0.0 159,81.86,96.46,79.315,79.46,79.17,72.35,0.0 160,81.93,96.56,79.055,79.02,79.09,72.78,0.0 161,81.20,96.13,78.81,79.29,78.33,72.61,0.0 162,81.29,96.27,78.61,78.16,79.06,72.56,0.0 163,81.58,96.21,79.025,78.64,79.41,72.74,0.0 164,81.54,96.16,78.86,79.32,78.4,72.74,0.0 165,81.84,96.43,79.11,79.27,78.95,72.97,1.0 166,81.52,96.38,78.835,79.02,78.65,72.82,0.0 167,82.19,96.43,79.34,79.43,79.25,72.65,0.0 168,82.26,96.38,79.35,79.43,79.27,72.71,0.0 169,81.19,96.34,78.175,78.6,77.75,73.10,0.0 170,81.81,96.49,78.815,79.07,78.56,72.92,0.0 171,82.41,96.63,79.765,79.74,79.79,73.05,0.0 172,82.01,96.27,78.76,78.05,79.47,73.07,0.0 173,81.75,96.48,78.935,79.17,78.7,72.68,0.0 174,81.72,96.45,79.115,79.59,78.64,73.22,0.0 175,81.91,96.48,79.4,79.42,79.38,73.72,0.0 176,82.01,96.69,79.275,79.32,79.23,73.40,0.0 177,81.98,96.43,79.035,79.18,78.89,73.49,0.0 178,82.47,96.44,79.79,79.39,80.19,73.02,0.0 179,82.16,96.25,79.265,79.67,78.86,73.22,0.0 180,82.11,96.50,79.58,79.83,79.33,73.54,1.0 181,81.97,96.41,78.85,78.67,79.03,73.73,0.0 182,81.57,96.27,79.01,78.77,79.25,73.01,0.0 183,81.80,96.62,78.865,78.56,79.17,73.48,0.0 184,81.53,96.51,78.83,78.73,78.93,73.27,0.0 185,81.94,96.28,79.1,79.12,79.08,73.53,0.0 186,83.01,96.38,80.245,80.25,80.24,73.49,1.0 187,80.86,96.06,77.705,78.39,77.02,73.53,0.0 187,82.41,96.41,79.715,79.84,79.59,73.93,1.0 188,81.76,96.54,78.81,78.85,78.77,73.76,0.0 189,82.39,96.48,79.405,79.34,79.47,73.36,0.0 190,81.75,96.25,78.745,78.77,78.72,73.97,0.0 191,81.96,96.33,78.87,79.16,78.58,73.59,0.0 192,81.43,96.29,78.305,77.56,79.05,73.63,0.0 193,81.79,96.18,78.985,79.09,78.88,73.25,0.0 194,82.15,96.44,79.19,79.74,78.64,74.06,0.0 195,81.60,96.30,78.74,79.11,78.37,73.79,1.0 196,82.20,96.29,79.06,79.,091.2,74.03,0.0 197,81.87,96.30,79.15,79.47,78.83,74.02,0.0 198,82.25,96.34,79.035,79.26,78.81,73.86,0.0 199,82.12,96.60,79.22,78.77,79.67,73.79,0.0 200,81.95,96.43,79.365,79.55,79.18,73.37,0.0 201,84.75,97.04,82.79,82.89,82.69,78.14,1.0 202,84.89,97.06,82.975,83.3,82.65,79.53,1.0 203,85.18,97.18,83.095,83.34,82.85,79.61,1.0 204,85.15,97.23,83.25,83.68,82.82,80.14,0.0 205,85.01,97.26,83.23,83.32,83.14,80.38,0.0 206,85.11,97.30,83.245,83.46,83.03,80.88,0.0 207,85.01,97.24,83.42,83.87,82.97,80.95,0.0 208,85.09,97.13,83.49,83.62,83.36,80.67,0.0 209,85.36,97.29,83.555,83.84,83.27,80.99,1.0 210,85.30,97.33,83.495,83.64,83.35,80.75,1.0 211,85.34,97.32,83.4,83.57,83.23,81.07,0.0 212,85.33,97.30,83.52,83.67,83.37,81.07,0.0 213,85.38,97.26,83.68,83.8,83.56,81.46,1.0 214,85.56,97.35,83.715,83.84,83.59,81.52,1.0 215,85.49,97.27,83.73,83.86,83.6,81.49,0.0 216,85.41,97.33,83.635,83.79,83.48,81.62,0.0 217,85.42,97.41,83.54,83.58,83.5,81.49,0.0 218,85.34,97.28,83.255,83.38,83.13,81.64,0.0 219,85.51,97.19,83.8,84.01,83.59,81.95,0.0 220,85.34,97.28,83.675,83.91,83.44,81.84,0.0 221,85.43,97.32,83.8,84.03,83.57,82.01,0.0 222,85.31,97.42,83.51,83.79,83.23,81.56,0.0 223,85.42,97.35,83.62,83.83,83.41,81.70,0.0 224,85.32,97.28,83.48,83.67,83.29,81.99,0.0 225,85.52,97.25,83.8,83.83,83.77,82.13,1.0 226,85.60,97.25,83.62,83.86,83.38,82.58,1.0 227,85.51,97.28,83.95,84.18,83.72,82.29,0.0 228,85.60,97.40,83.875,84.12,83.63,82.64,1.0 229,85.72,97.40,83.955,84.09,83.82,82.52,1.0 230,85.67,97.38,83.89,84.12,83.66,82.51,0.0 231,85.52,97.30,83.65,83.81,83.49,82.64,0.0 232,85.76,97.48,84.02,84.25,83.79,82.79,1.0 233,85.39,97.40,83.8,83.97,83.63,82.57,0.0 234,85.57,97.38,83.835,84.07,83.6,82.47,0.0 235,85.48,97.26,83.89,84.21,83.57,82.55,0.0 236,85.73,97.29,83.825,83.97,83.68,83.15,0.0 237,85.65,97.38,83.735,83.95,83.52,82.77,0.0 238,85.69,97.35,84.065,84.24,83.89,82.48,0.0 239,85.78,97.36,83.835,84.01,83.66,82.64,1.0 240,85.65,97.44,83.95,84.2,83.7,82.99,1.0 241,85.50,97.34,83.68,83.86,83.5,82.71,0.0 242,85.58,97.32,83.95,84.17,83.73,82.77,0.0 243,85.62,97.37,83.92,84.06,83.78,82.60,0.0 244,85.79,97.38,84.02,84.17,83.87,82.73,1.0 245,85.75,97.30,83.945,84.17,83.72,82.94,0.0 246,85.69,97.40,84.04,84.19,83.89,82.47,0.0 247,85.75,97.33,84.04,84.33,83.75,82.55,0.0 248,85.72,97.34,83.995,84.28,83.71,82.56,0.0 249,85.56,97.31,84.075,84.37,83.78,82.73,0.0 250,85.62,97.22,83.7,83.84,83.56,82.67,0.0 251,85.70,97.34,84.0,84.18,83.82,82.64,0.0 252,85.53,97.32,83.88,84.13,83.63,82.75,0.0 253,85.60,97.42,83.935,84.11,83.76,83.00,0.0 254,85.67,97.32,83.98,84.18,83.78,82.69,0.0 255,85.60,97.30,83.83,84.03,83.63,82.57,1.0 256,85.64,97.35,83.97,84.21,83.73,82.74,0.0 257,85.73,97.31,83.85,84.15,83.55,83.08,0.0 258,85.69,97.33,83.88,84.2,83.56,82.91,0.0 259,85.62,97.37,83.925,84.12,83.73,82.86,0.0 260,85.61,97.32,83.88,84.2,83.56,82.84,0.0 261,85.70,97.30,83.945,84.2,83.69,82.87,0.0 262,85.82,97.44,84.12,84.31,83.93,82.92,1.0 263,85.72,97.26,83.82,84.09,83.55,82.46,0.0 264,85.70,97.40,84.07,84.29,83.85,83.01,0.0 265,85.58,97.33,83.86,84.19,83.53,83.12,0.0 266,85.65,97.31,83.89,84.04,83.74,83.01,0.0 267,85.63,97.30,83.69,83.86,83.52,82.85,0.0 268,85.69,97.39,84.08,84.38,83.78,82.85,0.0 269,85.63,97.39,84.1,84.33,83.87,82.92,0.0 270,85.62,97.39,83.91,84.07,83.75,82.74,1.0 271,85.69,97.37,83.785,84.08,83.49,83.20,0.0 272,85.62,97.42,83.91,84.14,83.68,82.59,0.0 273,85.92,97.36,84.085,84.29,83.88,82.97,1.0 274,85.66,97.37,83.77,84.06,83.48,83.13,0.0 275,85.75,97.38,84.085,84.23,83.94,82.84,0.0 276,85.89,97.35,83.85,84.14,83.56,83.18,0.0 277,85.80,97.39,84.01,84.11,83.91,83.33,0.0 278,85.70,97.33,84.025,84.3,83.75,83.52,0.0 279,85.71,97.33,84.1,84.39,83.81,83.86,0.0 280,85.79,97.27,84.0,84.3,83.7,84.30,0.0 281,85.82,97.37,84.03,84.28,83.78,84.29,0.0 282,85.85,97.31,84.13,84.43,83.83,84.52,0.0 283,85.76,97.38,84.09,84.33,83.85,85.12,0.0 284,85.76,97.34,84.055,84.43,83.68,85.20,0.0 285,85.71,97.30,84.02,84.25,83.79,85.27,1.0 286,85.76,97.30,84.01,84.25,83.77,85.75,0.0 287,85.74,97.27,84.075,84.28,83.87,86.15,0.0 288,85.83,97.24,84.045,84.22,83.87,85.92,0.0 289,85.71,97.40,84.205,84.28,84.13,86.27,0.0 290,85.78,97.31,84.165,84.39,83.94,86.78,0.0 291,85.78,97.31,84.05,84.29,83.81,86.95,0.0 292,85.62,97.28,83.96,84.18,83.74,87.18,0.0 293,85.59,97.34,83.97,84.14,83.8,87.49,0.0 294,85.59,97.33,84.,041.3,83.87,87.65,0.0 295,85.55,97.30,83.84,83.99,83.69,87.79,0.0 296,85.63,97.35,84.05,84.18,83.92,88.07,0.0 297,85.77,97.35,83.89,84.16,83.62,88.40,0.0 298,85.67,97.38,83.96,84.05,83.87,88.54,0.0 299,85.61,97.26,83.9,84.05,83.75,88.85,0.0 300,85.63,97.25,83.825,83.98,83.67,89.03,1.0