search

TUI-NICR / ESANet

ESANet: Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis
Other
236 stars 51 forks source link

Training loss, validation loss and mIOU curves #20

Closed Bryan-Rathos closed 3 years ago

Bryan-Rathos commented 3 years ago

I had short question. Do you guys have loss curves for training and validation as well as mIOU curves with respect to epochs? I am using a custom dataset with equal distribution of classes in the training and validation set. I just have 4 classes but there there are certain classes which appear more compared to others. The distribution is close to 74%,11%,12%,3%. I've noticed that my validation decreases and reaches its lowest around 15% of the total epochs. However after that it starts increasing slightly and stays around the same range but the mIOU on the whole shows an upward trend. I just wanted to check if you have loss and miou curves to have some comparison. Thank you. Great job with ESANet! This shouldn't be included in the issues section, but, I couldn't find a discussion section to post this. Best, Bryan

danielS91 commented 3 years ago

Your question is quite complex to answer as the validation loss curve depends on many factors. First, we must distinguish between weighted and unweighted loss (see class weighing parameter. Due the weighting, the loss curve may look different, especially when the classes are heavily unbalanced. Second, the loss curve depends on the dataset and its consistency. For NYUv2 and SunRGB-D, we observed an increasing validation/test loss while the mIoU was increasing as well. We assume that this is due to label inconsistency. If the predictions get more “uncertain”, the loss increases but the argmax may remain the same or switches to another class. So, it is difficult to compare trends in mIoU and loss.

Bryan-Rathos commented 3 years ago

@danielS91 Thank you for your insightful response. The loss curves I mentioned was using the median frequency weighting. According to you, which would be the best checkpoint to use for inference? The one with the lowest validation loss (not the best mIoU) or the one with the highest mIoU? Looks like it is based on finding a trade off for the specific application. Based on your experience, how does the median frequency weighting perform in comparison to the logarithmic weighting?

danielS91 commented 3 years ago

For application, we usually choose the best checkpoint based on the mIoU. In most of our experiments median frequency weighting and logarithmic weighting performed similarly. However, for Cityscapes both performed significantly worse.

Bryan-Rathos commented 3 years ago

Thank you so much @danielS91 for your feedback. Closing this issue.