search

Joker316701882 / Additive-Margin-Softmax

This is the implementation of paper <Additive Margin Softmax for Face Verification>
492 stars 149 forks source link

Training time suddenly increases at some point? #15

Closed RemovedAccordingToRegualtions closed 6 years ago

RemovedAccordingToRegualtions commented 6 years ago

Epoch: [0][82/87866] Time 0.469 Loss 22.866 RegLoss 0.000000 Epoch: [0][83/87866] Time 0.468 Loss 22.917 RegLoss 0.000000 Epoch: [0][84/87866] Time 0.491 Loss 22.838 RegLoss 0.000000 Epoch: [0][85/87866] Time 0.453 Loss 22.843 RegLoss 0.000000 Epoch: [0][86/87866] Time 0.459 Loss 22.799 RegLoss 0.000000 Epoch: [0][87/87866] Time 0.459 Loss 22.761 RegLoss 0.000000 Epoch: [0][88/87866] Time 0.473 Loss 22.805 RegLoss 0.000000 Epoch: [0][89/87866] Time 0.466 Loss 22.893 RegLoss 0.000000 Epoch: [0][90/87866] Time 0.456 Loss 22.780 RegLoss 0.000000 Epoch: [0][91/87866] Time 0.457 Loss 22.721 RegLoss 0.000000 Epoch: [0][92/87866] Time 0.484 Loss 22.789 RegLoss 0.000000 Epoch: [0][93/87866] Time 0.465 Loss 22.911 RegLoss 0.000000 Epoch: [0][94/87866] Time 0.470 Loss 22.807 RegLoss 0.000000 Epoch: [0][95/87866] Time 0.483 Loss 22.808 RegLoss 0.000000 Epoch: [0][96/87866] Time 0.459 Loss 22.879 RegLoss 0.000000 Epoch: [0][97/87866] Time 0.459 Loss 22.673 RegLoss 0.000000 Epoch: [0][98/87866] Time 0.457 Loss 22.901 RegLoss 0.000000 Epoch: [0][99/87866] Time 0.494 Loss 22.780 RegLoss 0.000000 Epoch: [0][100/87866] Time 0.471 Loss 22.781 RegLoss 0.000000 Epoch: [0][101/87866] Time 0.458 Loss 22.840 RegLoss 0.000000 Epoch: [0][102/87866] Time 0.480 Loss 22.848 RegLoss 0.000000 Epoch: [0][103/87866] Time 0.470 Loss 22.788 RegLoss 0.000000 Epoch: [0][104/87866] Time 0.461 Loss 22.811 RegLoss 0.000000 Epoch: [0][105/87866] Time 0.471 Loss 22.855 RegLoss 0.000000 Epoch: [0][106/87866] Time 0.467 Loss 22.886 RegLoss 0.000000 Epoch: [0][107/87866] Time 0.453 Loss 22.856 RegLoss 0.000000 Epoch: [0][108/87866] Time 0.462 Loss 22.899 RegLoss 0.000000 Epoch: [0][109/87866] Time 0.490 Loss 22.719 RegLoss 0.000000 Epoch: [0][110/87866] Time 0.460 Loss 22.970 RegLoss 0.000000 Epoch: [0][111/87866] Time 0.457 Loss 22.761 RegLoss 0.000000 Epoch: [0][112/87866] Time 0.468 Loss 22.725 RegLoss 0.000000 Epoch: [0][113/87866] Time 0.479 Loss 22.768 RegLoss 0.000000 Epoch: [0][114/87866] Time 0.462 Loss 22.871 RegLoss 0.000000 Epoch: [0][115/87866] Time 0.456 Loss 22.947 RegLoss 0.000000 Epoch: [0][116/87866] Time 0.472 Loss 22.737 RegLoss 0.000000 Epoch: [0][117/87866] Time 0.472 Loss 22.790 RegLoss 0.000000 Epoch: [0][118/87866] Time 0.454 Loss 22.665 RegLoss 0.000000 Epoch: [0][119/87866] Time 0.453 Loss 22.742 RegLoss 0.000000 Epoch: [0][120/87866] Time 1.965 Loss 22.788 RegLoss 0.000000 Epoch: [0][121/87866] Time 1.992 Loss 22.809 RegLoss 0.000000 Epoch: [0][122/87866] Time 1.956 Loss 22.791 RegLoss 0.000000 Epoch: [0][123/87866] Time 2.076 Loss 22.890 RegLoss 0.000000 Epoch: [0][124/87866] Time 2.076 Loss 22.677 RegLoss 0.000000 Epoch: [0][125/87866] Time 2.029 Loss 22.981 RegLoss 0.000000 Epoch: [0][126/87866] Time 2.021 Loss 22.807 RegLoss 0.000000 Epoch: [0][127/87866] Time 1.953 Loss 22.661 RegLoss 0.000000 Epoch: [0][128/87866] Time 1.979 Loss 22.750 RegLoss 0.000000 Epoch: [0][129/87866] Time 2.029 Loss 22.915 RegLoss 0.000000

RemovedAccordingToRegualtions commented 6 years ago

Im running the code on GTX1080Ti. You can see there is a significant increase from 0.4 to 2.0 around 119 and 120. I checked the code nothing suspicious. Am I missing something?

RemovedAccordingToRegualtions commented 6 years ago

I ran it several times with the same set of parameters. The performance drop happens quite randomly sometimes it happens at 50, sometimes 200.

Joker316701882 commented 6 years ago

@RemovedAccordingToRegualtions Try it again. I just fix some bugs.