Increase in inference time after fine-tuning

[bhavitvyamalik]

Hi! I fine-tuned your pedestrian model on my custom pedestrian dataset and it's performing really well now but its inference time has increased a bit. Earlier I was getting 9.7 FPS (I'm using 5W mode on Jetson Nano) and now I'm getting around 9 FPS. Why is it so?

Also, is there any way to see if my model has overfit or it is still learning? My losses were too high. Here's the loss log:

2020-05-27 07:55:44,698[INFO]: Start validating-------------------------------------------
2020-05-27 07:55:45,693[INFO]: Iter[200000] validation metric ------------- 
2020-05-27 07:55:45,693[INFO]: CE_loss_score_0: --> 407.6001
2020-05-27 07:55:45,694[INFO]: SE_loss_bbox_0: --> 808.5334
2020-05-27 07:55:45,694[INFO]: CE_loss_score_1: --> 516.9276
2020-05-27 07:55:45,705[INFO]: SE_loss_bbox_1: --> 630.4637
2020-05-27 07:55:45,713[INFO]: CE_loss_score_2: --> 557.3298
2020-05-27 07:55:45,715[INFO]: SE_loss_bbox_2: --> 549.5728
2020-05-27 07:55:45,715[INFO]: CE_loss_score_3: --> 220.4552
2020-05-27 07:55:45,720[INFO]: SE_loss_bbox_3: --> 567.7109
2020-05-27 07:55:45,726[INFO]: End validating ---------------------------------------------

[YonghaoHe]

@bhavitvyamalik I think the inference time gap is reasonable for Jetson Nano. The losses seem not low enough.

[bhavitvyamalik]

I trained this model for 300000 steps and still not much difference was seen in losses. How were the results when you used Adam optimizer instead of SGD which you are using currently?

[YonghaoHe]

@bhavitvyamalik Adam may not give you a good convergence, use SDG instead.

[RamatovInomjon]

Hi @YonghaoHe ! I'm trying to run your head detection model on Jetson nano, it could run only 480p and scale=0.25 , I couldn't understand reason , please guide