Open hashJoe opened 9 months ago
@hashJoe Hey, for me the loss_bbox and loss_giou are not decresing, but rather constant. Do you have an idea what the reason could be? My train_loss is going down and the metrics up.
How big is your dataset?
@hashJoe Hey, for me the loss_bbox and loss_giou are not decresing, but rather constant. Do you have an idea what the reason could be? My train_loss is going down and the metrics up.
How big is your dataset?
@amndzr Around 5K images and maybe 13K boxes. Losses are going down and results are good. Are your results still looking good?
Try debugging into this and check what's wrong. https://github.com/lyuwenyu/RT-DETR/blob/94f5e16708329d2f2716426868ec89aa774af016/rtdetr_pytorch/src/zoo/rtdetr/rtdetr_criterion.py#L152-L171
@hashJoe I cant seem to find anything suspicious when debugging. My results are good in respect to train_loss, AP as well as AR. My Dataset has a size of around 10K images.
Hello, I have the same problem as you, do you know the cause of the problem, and how did you solve it?
@amndzr I see it is just the validation loss.. the same reply below might help:
@20231211 The transforms applied in validation stage on the bounding boxes differ than that of the training stage.
The training stage, the boxes coordinate output format are changed to cxcywh
and normalized as well:
https://github.com/lyuwenyu/RT-DETR/blob/64878acad2f58ed34579e5a5ec45da1044587e09/rtdetr_pytorch/src/data/transforms.py#L132-L139
where as in the validation stage, this fact is ignored and therefore unnormalized, which result in wrong scaling to compute in the criterion. https://github.com/lyuwenyu/RT-DETR/blob/64878acad2f58ed34579e5a5ec45da1044587e09/rtdetr_pytorch/configs/rtdetr/include/dataloader.yml#L35
@lyuwenyu Could you please check this? Thanks! :)
@hashJoe Thank you for your help, so I should replace {type: ConvertDtype} with {type: ConvertBox, out_fmt: 'cxcywh', normalize: True}, right? Do I need to make any other changes?
@20231211 That's what I did, then the criterion called here in: https://github.com/lyuwenyu/RT-DETR/blob/64878acad2f58ed34579e5a5ec45da1044587e09/rtdetr_pytorch/src/solver/det_engine.py#L123 will compare scaled output boxes coordinates similar to the model output as in https://github.com/lyuwenyu/RT-DETR/blob/64878acad2f58ed34579e5a5ec45da1044587e09/rtdetr_pytorch/src/zoo/rtdetr/rtdetr_decoder.py#L555 If I am not mistaken .. and model doesn't treat boxes differently in validation than in training
@hashJoe Thank you so much!
@20231211 No I didnt continue working on it. However, I trained the model not on my custom dataset but on the COCO2017 Dataset to see if the loss is still consistant. What I found was, that the validation Loss was still consistant, but when testing the model on unseen data the results were good so I didnt bother.
@amndzr I tried the above workaround, adding {type: ConvertBox, out_fmt: 'cxcywh', normalize: True}, and the test loss drops and is on an order of magnitude higher than the training loss.
Hey, I inserted criterion losses into
metric_logger
to have a better overview of the evaluation besidescoco_eval_bbox
. I simply uncommented the lines referenced below and returned the losses.https://github.com/lyuwenyu/RT-DETR/blob/3330eca679a7d7cce16bbb10509099174a2f40bf/rtdetr_pytorch/src/solver/det_engine.py#L123-L133
However, the values of the losses differ than those of the training losses. Below are examples of train and test losses of the same epoch:
e.g., Train
Test/Val
loss_vfl
seem to be alright, butloss_bbox
andloss_giou
are much larger, but decreasing in values as the model is trained for more epochs.Is this an issue? What's the reason behind this? Thank you!