SkyTruth / cerulean-ml

Repo for Training ML assets for Cerulean
Apache License 2.0
1 stars 0 forks source link

Address icevision 3-band training accuracy #70

Open rbavery opened 2 years ago

rbavery commented 2 years ago

On a first pass through 10 epochs of the 3 band dataset, accuracy of the IOU metric is low, with no predictions being generated for any example results. This is in contrast to the 1 band dataset which does generate some valid predictions on a smaller partial dataset.

I'll be looking into the trainer more closely to see why no valid predictions are being generated. First thing I'll do is to increase the resolution of the train and loss curves and try to substantially overfit on the training set, since currently training loss decreases at about the same rate as the validation set.

rbavery commented 2 years ago

List of priorities for improving Mask RCNN accuracy:

first thing I'll try is comparing the simpler fastai unet 1 band to 3 band to see if this is an issue with the aux data or the icevision trainer.

rbavery commented 2 years ago

Update, the fastai unet Dice loss goes up to .3 when training on the first progressive resize of 64. this is slightly above the Dice loss of .2 after 5 epochs for CV1. so it would appear that there's something specific to the icevision implementation that is causing problems. @lillythomas has also discovered a new error where with the single band model, training cuases kernel death after about 4 epochs. We're tabling these issues while we focus on the Unet baseline.