joonson / syncnet_trainer

Disentangled Speech Embeddings using Cross-Modal Self-Supervision
MIT License
154 stars 26 forks source link

Model overfitting when finetuned on smaller data #20

Open diksham-flwls opened 9 months ago

diksham-flwls commented 9 months ago

Hi I am trying to finetune the model on a smaller dataset of 450 samples of greyhead renders that look like this

Screenshot 2024-02-06 at 10 42 08 AM

My training loss seems to be converging but the validation loss is diverging. I am freezing everything but the final layers but otherwise using the exact same code as the repo.

Screenshot 2024-02-07 at 7 41 56 PM Screenshot 2024-02-07 at 7 42 06 PM

When I plot the euclidean distance of the FC layer outputs I find that the distribution is pretty overlapping for matched (green) and unmatched pairs (red). matched_and_unmatched_pair_dist