Fine-tune pretrained model with own datasets, get 0 precision, recall, F1, iou after a few epochs

[Plan-T42]

Hi, I am following your tips in the closed issue #42 Best Model: Load the entire weights file you shared in repo; freeze the decoder part; then start training on my own dataset.

However, after a few epochs, the accuracy reached pretty high, but precision, F1, recall, and iou got to zero, in both training and validation metrics.

I tried to use my finetuned model to make inferences - apparently it predicts all the pixels of a test image as non-buildings.

I suspect it is because of the imbalance dataset I have - The buildings are rather sparse. But I checked the Ineria dataset you used, they are not always intensive with buildings either.

I am wondering what tricks you have used to tackled this, or do you have any suggestions?

Thanks in advance! And great work you have done!

[fuzailpalnak]

Will it be possible to share a small subset of your dataset ?

[Plan-T42]

This is the whole dataset I am using now. I'm also wondering whether the amount is too small for proper fine tuning.

https://drive.google.com/drive/folders/1OHJwMPWwQAjxEybxA5KYFiMOUZqG-Qjp?usp=share_link

And as reference my codes are in this Colab Notebook.

https://colab.research.google.com/drive/1g42I9GwQ2zEe5Dtp6dV-wBCs9ETc29Oz?usp=sharing

(Both links need access permits, just send the requst, I will grant. )

Thanks!

[fuzailpalnak]

The dataset size is too small to run it on RefineNet, that’s what I suspect, when I turned off all the layers and just trained on the last layer the network seems to generate some blobs like structure.

If you have such a small dataset with no access to additional data, then I would suggest is to use RefineNet to extract features as it seems to give good predictions on the data you provided and a feed that to a small neural network it can be a fully connected layer and then check what results you get.

You could also take another approach and use augmentations to generate synthetic data and increase the size of the dataset. One thing to watch out for is, generalisation , the model could very well just generalise to synthetic data and not to the original data and this would lead to problems.

[Plan-T42]

Thanks a lot!

How large the size of the dataset should be to properly serve the fine-tuning approach in your opinion? And did you actually freeze more than the decoder layers?

"Use RefineNet as a feature extractor and a feed that to a small neural network" - Do you possibly have suggestions on up to which layers of RefineNet to freeze and what architecture of the "small neural network" as the final classifier to use?

Yeah I will give a try for data augmentation too! So you don't think there's any unbalance problem of my dataset? (As if that's the case, I will augment more on the building-intensive area for example)

[fuzailpalnak]

[1] I don’t have an exact number, but I would say something above 3K. I just trained the final layer of the decoder, rest were all set to trainable=False, including the rest of the decoder.

[2] For feature extraction part this is a trail and error process, To begin with I would start with the output of the final layer, i.e before the model maps it to class, and use those to train an SVM or something similar and observe the output and proceed accordingly. I would also experiment with adding an Dense layer just the before class mapping and observe the network’s performance.

[3] This kinda of imbalance is pretty common. What I can suggest is you remove all the images which don’t have any building labels in them. And consider using IOU as your metric for evaluation rather than Accuracy. Consider using weighted loss function Issue , you can experiment with loss function [dice loss, jaccard loss, …] that addresses this kind of issue.

Hope this helps