senarvi
commented
8 years ago The problem is probably caused by batch iterator not returning to the same position in the file, when recalling a previous training state.
_set_candidate_state
[301036] Validation set cost history: 1164.4 1088.1 1040.3 1034.5 1010.8 1013.9 1012.4 1015.2 999.3 1002.3 [963.7]
[302000] (55.18 %) of epoch 1 -- lr = 0.001, cost = 2.07, duration = 4.77 ms
[547000] (99.94 %) of epoch 1 -- lr = 0.001, cost = 2.23, duration = 4.75 ms
[547337] Validation set cost history: 1164.4 1088.1 1040.3 1034.5 1010.8 1013.9 1012.4 1015.2 999.3 1002.3 [963.7] 997.1 986.8 995.3 984.1 986.2 986.7 994.1 984.8 987.9
[663] (0.12 %) of epoch 2 -- lr = 0.001, cost = 1.78, duration = 6.80 ms
[54663] (9.99 %) of epoch 2 -- lr = 0.001, cost = 2.05, duration = 4.35 ms
[54734] Validation set cost history: 1040.3 1034.5 1010.8 1013.9 1012.4 1015.2 999.3 1002.3 [963.7] 997.1 986.8 995.3 984.1 986.2 986.7 994.1 984.8 987.9 987.6 975.6
reset_state
[301031] Validation set cost history: 1164.4 1088.1 1040.3 1034.5 1010.8 1013.9 1012.4 1015.2 999.3 1002.3 [963.7]
[301960] (55.17 %) of epoch 1 -- lr = 0.0005, cost = 1.59, duration = 7.60 ms
[409960] (74.90 %) of epoch 1 -- lr = 0.0005, cost = 2.69, duration = 3.23 ms
[410503] Validation set cost history: 1164.4 1088.1 1040.3 1034.5 1010.8 1013.9 1012.4 1015.2 999.3 1002.3 963.7 962.8 975.3 966.4 958.3 [948.9]
[410960] (75.08 %) of epoch 1 -- lr = 0.0005, cost = 1.93, duration = 3.11 ms
[546960] (99.93 %) of epoch 1 -- lr = 0.0005, cost = 1.86, duration = 4.41 ms
[547337] Validation set cost history: 1088.1 1040.3 1034.5 1010.8 1013.9 1012.4 1015.2 999.3 1002.3 963.7 962.8 975.3 966.4 958.3 [948.9] 964.9 957.2 953.2 966.3 954.8
[547960] (100.11 %) of epoch 1 -- lr = 0.0005, cost = 1.54, duration = 6.42 ms
Epoch and update numbers are not set correctly anymore in the end of an epoch? Progress counter goes over 100 % of epoch 1.