Early stopping a bit aggressive for low number of epochs

[cwmeijer]

According to https://mcfly.readthedocs.io/en/latest/reference.html?highlight=early#mcfly.train_models_on_samples, Unless ‘None’ early Stopping is used for the model training. Set to integer to define how many epochs without improvement to wait for before stopping. Default is ‘auto’ in which case the patience will be set to number of epochs/10 (and not bigger than 5). This means that if we select a number of epochs below 10, by default, any failure to improve within 1 epoch will trigger early stopping. As a default, this is too aggressive. Alternatively, we don't do early stopping if the total number of epochs is below 10.

To illustrate see:

Training model 0 CNN
Epoch 1/4
107/107 [==============================] - 26s 217ms/step - loss: 1.5502 - accuracy: 0.8583 - val_loss: 2.3463 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 23s 213ms/step - loss: 1.0574 - accuracy: 0.8705 - val_loss: 2.5239 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 1 CNN
Epoch 1/4
107/107 [==============================] - 25s 191ms/step - loss: 0.6182 - accuracy: 0.8681 - val_loss: 0.9670 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 20s 190ms/step - loss: 0.5402 - accuracy: 0.9000 - val_loss: 0.9284 - val_accuracy: 0.5229
Epoch 2: early stopping
Training model 2 CNN
Epoch 1/4
107/107 [==============================] - 18s 144ms/step - loss: 14.1103 - accuracy: 0.8508 - val_loss: 5.0653 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 15s 136ms/step - loss: 2.9834 - accuracy: 0.8545 - val_loss: 2.2986 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 3 CNN
Epoch 1/4
107/107 [==============================] - 8s 57ms/step - loss: 0.5379 - accuracy: 0.8756 - val_loss: 12.6588 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 6s 55ms/step - loss: 0.4464 - accuracy: 0.8846 - val_loss: 5.0486 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 4 CNN
Epoch 1/4
107/107 [==============================] - 14s 115ms/step - loss: 5.6878 - accuracy: 0.8447 - val_loss: 6.2826 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 12s 110ms/step - loss: 1.0738 - accuracy: 0.8555 - val_loss: 1.9365 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 5 CNN
Epoch 1/4
107/107 [==============================] - 7s 47ms/step - loss: 3.2859 - accuracy: 0.8634 - val_loss: 12.3406 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 5s 42ms/step - loss: 1.5800 - accuracy: 0.8710 - val_loss: 2.6766 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 6 CNN
Epoch 1/4
107/107 [==============================] - 15s 110ms/step - loss: 2.0351 - accuracy: 0.8630 - val_loss: 5.8495 - val_accuracy: 0.5187
Epoch 2/4
107/107 [==============================] - 10s 96ms/step - loss: 1.2047 - accuracy: 0.8794 - val_loss: 4.2294 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 7 CNN
Epoch 1/4
107/107 [==============================] - 16s 106ms/step - loss: 0.4598 - accuracy: 0.8738 - val_loss: 1.1363 - val_accuracy: 0.4813
Epoch 2/4
107/107 [==============================] - 10s 91ms/step - loss: 0.3988 - accuracy: 0.8911 - val_loss: 0.9418 - val_accuracy: 0.5187
Epoch 2: early stopping
Training model 8 CNN

All of the above models are decreasing their loss on the validation set, but did not have the time yet to switch to any better strategy than choosing the majority class in the train set.