search

ZheC / Realtime_Multi-Person_Pose_Estimation

Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)
Other
5.08k stars 1.36k forks source link

training convergence issue #126

Open coldgemini opened 6 years ago

coldgemini commented 6 years ago

when I train from scratch except for the VGG model part, the loss is always around 1500 and not going any lower after one night training in one GPU. Is this the normal case, since usually the loss drop faster initially then stops somewhere?

ZheC commented 6 years ago

This might be helpful: https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation/issues/19

Ai-is-light commented 6 years ago

@coldgemini Did you get the same results as the paper mentioned? Thanks

qiujing27 commented 6 years ago

@ZheC Hi, I followed the exact code to train but still can't get the reported accuracy. The training data I am using is generated from your code (around 202.5GB instead of the link's 189GB data). I didn't change any parameter defined in the setLayers.py and used vgg pretrained model. The final mAP(IoU=05:0.95) is around 0.53 instead of 0.58 for my model on iteration 440000 and iteration 566000. May I know did you use other data augmentation strategy or different training parameters to get the posted caffe model (pose_iter_440000.caffemodel)?

Ai-is-light commented 6 years ago

@qiujing27 would you mind sharing you train log or loss? and how long did you cost?thanks

qiujing27 commented 6 years ago

Sure, it took me totally around 8 days for finishing training at iteration 566000, it was run on one Titan Xp and took 40 -50 mins for training every 2000 iterations. However, my training log before 350000 iterations seemed to be was overwritten and could not be found now.....

2018-01-11 0:55 GMT-08:00 Yong notifications@github.com:

@qiujing27 https://github.com/qiujing27 would you mind sharing you train log or loss? and how long did you cost?thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ZheC/Realtime_Multi-Person_Pose_Estimation/issues/126#issuecomment-356867773, or mute the thread https://github.com/notifications/unsubscribe-auth/AYI-xnA97jyuXYnwfcFdclwvk8ERv-s1ks5tJcx1gaJpZM4Q5fJb .

Ai-is-light commented 6 years ago

@ @qiujing27 how about the final loss you got in your practice

Ai-is-light commented 6 years ago

what's the final loss values? @qiujing27