search

bradyz / cross_view_transformers

Cross-view Transformers for real-time Map-view Semantic Segmentation (CVPR 2022 Oral)
MIT License
525 stars 80 forks source link

Val IoU not good after training #32

Closed ArminBaz closed 10 months ago

ArminBaz commented 2 years ago

Hey there, thank you for your great contribution! Unfortunately I am running into some issues when I am train the model using the nuscenes vehicle experiment config.

The iou metrics are as follows: Train/Val iou@0.4 iou@0.5
Train 0.4305 0.3868
Val 0.09296 0.03948

So, you can see that there are some issues during validation?

Interestingly, this can also be visually confirmed as during training the models outputs get better and better. However, during validation the predictions get better, then there is a period where the model only predicts a black screen, and then they start getting better again.
Following are some photos from the w&b log: Val metrics plot image

Val output at step 247 image

Val output at step 329 (notice all black) image

Val output at step 663 image

Would really appreciate any feedback you might have regarding this issue. Thanks.

gsg213 commented 1 year ago

Hello @ArminBaz ,

Did you find the issue? Did you change something in the model architecture?

For further analysis, you could share the loss and lr graphs to understand better what happened there.

ArminBaz commented 1 year ago

Hey @gsg213, I did end up resolving the issue on my own.

What worked for me was lowering the learning rate inside of config.yaml from 4e-3 to 1e-3.

Hope this helps!