search

nmi-lab / decolle-public

GNU General Public License v3.0
41 stars 22 forks source link

Training Time #5

Open meltingCat opened 3 years ago

meltingCat commented 3 years ago

HI, I would like to know how many time did you train on an epoch. I spent an hour to train an epoch using RTX2080. Doesn't seem right to deal with such small dataset.

---------------Epoch 1------------- ---------Saving checkpoint--------- Testing: 100%|████████████████████████████████| 124/124 [03:28<00:00, 1.68s/it] Error Rate L0 0.315 Error Rate L1 0.0407 Error Rate L2 0.0249 Error Rate L3 0.0246 Epoch 1: 100%|████████████████████████████████| 753/753 [54:31<00:00, 4.34s/it] Loss [ 28.6935 6.8569 4.2865 895.1974] Activity Rate [95.05911542127708, 49.73174018810146, 38.44802363960695, 66.41492680630824] Changing learning rate from 1e-09 to 1e-09

eneftci commented 3 years ago

Which dataset is this? and what are the dimensions of your input [batch_size, timesteps, channels, height, width]? 4s per iteration seems reasonable.

meltingCat commented 3 years ago

I ran “train_lenet_decolle.py” with default setting and Mnist . Maybe I should increase the batch size, then try again.

eneftci commented 3 years ago

Unfortunately this is normal. N-MNIST is a bit long to train.