search

vacancy / Synchronized-BatchNorm-PyTorch

Synchronized Batch Normalization implementation in PyTorch.
MIT License
1.5k stars 189 forks source link

test gap between training and test #30

Open ZhiweiYan-96 opened 5 years ago

ZhiweiYan-96 commented 5 years ago

Hi~ Thanking for your code firstly !

I use the SyncBatch for training SSD, when training I can get 46.81% mAP after 10 epoches finishes. However, when I use the saved model, I only get 36.7% mAP

Here is how I convert model ssd_student = convert_model(ssd_student) net_student = ssd_student

This is code for saving model torch.save(net_student.state_dict(), 'weights/' + model_name + '/ssd300_COCO_' + repr(iteration) + '_%.2f.pth' % mAP_student)

When load saved model, I didnt use the convert_model to convert to SyncBN since i think there is no need to convert model when key actually is the same.

I couldn't find where I'm wrong, please give some kind advice, thank you !!

MJITG commented 5 years ago

Same question

vacancy commented 5 years ago

I am sorry that I am not currently actively using this tool. Can any of the users provide a minimal script for reproducing the train-test gap results? Appreciate!

phtu-cs commented 1 year ago

Thank you for your code contribution. The same thing happened to me. Maybe there are some bugs in keeping track of batch statistics.