develop binary classification model

[ebgoldstein]

use TF to make classification model ( i.e., a notebook).

[ebgoldstein]

here is a simple transfer learning template: https://github.com/ebgoldstein/Coastal_TF_templates/blob/main/src/Classification.ipynb

[ebgoldstein]

to do:

• [x] use mobileNetv2 (put pics training curves here in issue, and commit jupyter notebook to github)
• [x] show it works with gradcam (put gradcam example images here in the issue)

[Bolewitz]

Committed jupyter notebook, 9b973aa. Curves seem to look okay, but predictions using this model appear to be very wrong. Using gradcam it looks like the features it is picking out are wrong. Sometimes picking out cars or light hitting the lens in flooded images.

[Bolewitz]

Debugging resources https://distill.pub/2017/feature-visualization/ https://wandb.ai/site/experiment-tracking

[ebgoldstein]

• [ ] Add a single canonical example notebook where you are retraining only some number of classification layers, not the base model. Make sure the the notebook saves the model at the end
• [ ] Have a new notyebook that loads the model and makes prediction on a single image, a stack of images (from one class), and then the full set of images... make a confusion matrix in that notebook...
• [ ] update this issue with the problems...

[Bolewitz]

Committed new training and testing notebooks 6d10270, 3306b8f.

This model was trained using images from BF_01, CB_01, and CB_03. Models and this dataset can be found on the Google Drive. Some stats for other models I trained can also be found there.

Some notes and problems:

• The original "poor" performance may have been due to the validation set being shuffled. After some testing I discovered the labels and the images were not in the same order, causing the confusion matrix to be incorrect. So validation_generator has shuffle = False now.
• Loss and accuracy curves are worse than when I was training with just BF_01 images.
• Difficulties preventing over-fitting, and validation accuracy lagging behind training accuracy, eventually plateauing.
• In the committed ModelTester notebook, the model has many false positives.
• Working on integrating Weights&Biases, it appears to be free for academics.

[ebgoldstein]

1. great to see that the confusion matrix (and scores) now match the validation curves.. (i.e., model is ~70% right now)
2. This makes sense:

   Loss and accuracy curves are worse than when I was training with just BF_01 images.

3. Val accuracy plateauing is ok for now. I do think the next step is to try 'fine-tuning' the whole model, with learning rate set very low as a second step.. look at this example: https://keras.io/guides/transfer_learning/

I suggest that after you 'fine tune' the model, you recommit the notebook, close this issue, and we walk through the quantization steps.. (i.e., don;t worry about W&B for now)..

[Bolewitz]

Recommitted the notebook. a477ede Loaded up the same model, set the layers to trainable, and retrained for 30 epochs with a very low learning rate. The training curves made it seem like the model was not improved after fine tuning. However, the f1 score for both flooded and not flooded improved from the fine tuning. Model testing results after fine tuning:

          precision    recall  f1-score   support

   False       0.87      0.88      0.88      2365
    True       0.79      0.77      0.78      1356

accuracy                           0.84      3721
macro avg      0.83      0.83      0.83      3721
weighted avg   0.84      0.84      0.84      3721

This model has been uploaded to google drive as "mobilenetv2_water_18finetune".