thomashopkins32
commented
1 year ago After running some small experiments, I have noticed the following:
- Learning rate should start high and be decreased over time. Current start has been
1e-2. - Momentum should be very high 0.95-0.99 depending on batch size. The smaller the batch size, the larger the momentum.
- Data transformations should include elastic changes and image flipping at a minimum.
Looking to try the following next:
- Larger batch size (run on Kaggle or Colab)
- Increase learning rate until its no longer helpful
- Decrease momentum to
0.9 - Add back affine transformations for rotation/translation
- Try with and without dropout on the final layer of the network
- Try with and without "proper" initialization according to UNet paper
- Try cosine annealing and other schedulers. Maybe one based on validation mAP
- Try increasing the batch size while using automatic mixed precision (AMP) training
Follow Karpathy's recipe for training: http://karpathy.github.io/2019/04/25/recipe/
We want to settle on what is working and what isn't. I think its clear that release 1 and release 2 have overfit quite a bit to the training data.