search

sming256 / OpenTAD

OpenTAD is an open-source temporal action detection (TAD) toolbox based on PyTorch.
Apache License 2.0
106 stars 5 forks source link

Training loss drops sharply to Nan on my own dataset #8

Closed qinchenmeng closed 2 months ago

qinchenmeng commented 2 months ago

May I ask why my loss suddenly decreased to Nan during training using my own dataset? Looking forward to your answer,thanks。 image

sming256 commented 2 months ago

I recommend that you check the data and annotation first. If the input data is corrupted and has NaN, then the loss is wrong. Also, if a video does not have any ground truth, or the ground truth is wrong, such as the end time is smaller than the start time, it may also lead to an error.

If the above is OK, you can consider whether you need to tune the backbone learning rate, or maybe the detector has some problem, and you might need to change to another detector.

qinchenmeng commented 2 months ago

Thank you for your patient reply. I will check the above methods