Nan occur during training

[ZhenhuiTang]

Hi Weijing, I have some questions during training and evaluation. As shown below, the reg loss get nan from the first step. I just change the training dir path and set GPU and batch are 1 respectively.

[ZhenhuiTang]

Besides, i have some puzzle about the result of evaluation:

1. Dose the "car_dectection_AP" mean the AP (IOU=0.5) in Image plane?
2. what the "car_orientation_AOS" mean ?
3. Dose the "car_detection_BEV_AP"is with IOU=0.7?
4. Dose the "car_detection_3D_AP"is with IOU=0.7?
5. where is AOS ?
6. where is the evaluation result to cyclists and pedestrians?

best regards.

[WeijingShi]

Hi, Nan is common when the learning rate is too large. You are using batch_size=1 so the gradient is noisier and you might need a smaller learning rate. You can try to reduce it and see if you still get Nan. In my experiment, I also found gradient clipping and the cyclical learning rate are helpful.

Does the "car_detection_BEV_AP"is with IOU=0.7? yes. Does the "car_detection_3D_AP"is with IOU=0.7? yes. We do not use AOS (part of the AVOD paper) or the 2D AP (IOU=0.7). Please refer to asharakeh 's implementation for what they mean.

The cyclists and pedestrians are detected by a separated model

Thanks,

[ZhenhuiTang]

Hi, Nan is common when the learning rate is too large. You are using batch_size=1 so the gradient is noisier and you might need a smaller learning rate. You can try to reduce it and see if you still get Nan. In my experiment, I also found gradient clipping and the cyclical learning rate are helpful.

Does the "car_detection_BEV_AP"is with IOU=0.7? yes. Does the "car_detection_3D_AP"is with IOU=0.7? yes. We do not use AOS (part of the AVOD paper) or the 2D AP (IOU=0.7). Please refer to asharakeh 's implementation for what they mean.

The cyclists and pedestrians are detected by a separated model

Thanks,

I'm really grateful.