search

kevinzakka / recurrent-visual-attention

A PyTorch Implementation of "Recurrent Models of Visual Attention"
MIT License
468 stars 123 forks source link

Performance not as claimed #11

Closed ipod825 closed 4 years ago

ipod825 commented 6 years ago

I checkout the recent commit that changed the optimizer. Though you claim that "With the Adam optimizer, paper accuracy can be reached in 30 epochs." But as I run python main.py --is_train 1, the performance isn't so desirable as claimed. Here's the log of my running result. Can you confirm on this? (run based on commit 99c4cbe)

Epoch: 1/200 - LR: 0.000300
159.4s - loss: 0.484 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:39<00:00, 338.75it/s]
train loss: 1.646 - train acc: 44.728 - val loss: 0.898 - val acc: 73.050 [*]

Epoch: 2/200 - LR: 0.000300
136.2s - loss: 1.691 - acc: 62.500: 100%|████████████████████████| 54000/54000 [02:16<00:00, 396.43it/s]
train loss: 0.928 - train acc: 69.515 - val loss: 0.667 - val acc: 80.483 [*]

Epoch: 3/200 - LR: 0.000300
178.1s - loss: 1.099 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:58<00:00, 303.14it/s]
train loss: 0.754 - train acc: 77.141 - val loss: 0.255 - val acc: 89.717 [*]

Epoch: 4/200 - LR: 0.000300
164.8s - loss: 0.124 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:44<00:00, 327.69it/s]
train loss: 0.711 - train acc: 79.198 - val loss: 0.448 - val acc: 88.617

Epoch: 5/200 - LR: 0.000300
164.7s - loss: 1.203 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:44<00:00, 327.76it/s]
train loss: 0.711 - train acc: 79.774 - val loss: 0.218 - val acc: 91.150 [*]

Epoch: 6/200 - LR: 0.000300
152.3s - loss: 1.857 - acc: 62.500: 100%|████████████████████████| 54000/54000 [02:32<00:00, 354.63it/s]
train loss: 0.690 - train acc: 80.306 - val loss: 0.074 - val acc: 92.233 [*]

Epoch: 7/200 - LR: 0.000300
167.9s - loss: 0.470 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:47<00:00, 321.62it/s]
train loss: 0.644 - train acc: 81.596 - val loss: 0.187 - val acc: 92.150

Epoch: 8/200 - LR: 0.000300
160.9s - loss: 0.292 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:40<00:00, 335.51it/s]
train loss: 0.605 - train acc: 82.700 - val loss: 0.176 - val acc: 92.700 [*]

Epoch: 9/200 - LR: 0.000300
137.5s - loss: 1.000 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:17<00:00, 392.73it/s]
train loss: 0.590 - train acc: 83.144 - val loss: 0.179 - val acc: 93.017 [*]

Epoch: 10/200 - LR: 0.000300
157.0s - loss: 1.242 - acc: 81.250: 100%|████████████████████████| 54000/54000 [02:36<00:00, 407.21it/s]
train loss: 0.567 - train acc: 84.050 - val loss: 0.200 - val acc: 93.133 [*]

Epoch: 11/200 - LR: 0.000300
160.3s - loss: 1.275 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:40<00:00, 345.11it/s]
train loss: 0.544 - train acc: 84.524 - val loss: 0.182 - val acc: 95.033 [*]

Epoch: 12/200 - LR: 0.000300
173.2s - loss: 1.563 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:53<00:00, 242.64it/s]
train loss: 0.536 - train acc: 84.783 - val loss: 0.192 - val acc: 94.417

Epoch: 13/200 - LR: 0.000300
155.8s - loss: 0.343 - acc: 93.750: 100%|████████████████████████| 54000/54000 [02:35<00:00, 346.46it/s]
train loss: 0.525 - train acc: 85.424 - val loss: 0.127 - val acc: 95.217 [*]

Epoch: 14/200 - LR: 0.000300
159.2s - loss: 0.462 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:39<00:00, 339.10it/s]
train loss: 0.530 - train acc: 85.400 - val loss: 0.139 - val acc: 94.967

Epoch: 15/200 - LR: 0.000300
162.9s - loss: 0.065 - acc: 93.750: 100%|████████████████████████| 54000/54000 [02:42<00:00, 331.49it/s]
train loss: 0.525 - train acc: 85.461 - val loss: 0.110 - val acc: 94.983

Epoch: 16/200 - LR: 0.000300
173.3s - loss: 0.422 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:53<00:00, 345.51it/s]
train loss: 0.553 - train acc: 84.639 - val loss: 0.208 - val acc: 94.400

Epoch: 17/200 - LR: 0.000300
140.1s - loss: 0.626 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:20<00:00, 385.36it/s]
train loss: 0.555 - train acc: 84.563 - val loss: 0.196 - val acc: 95.383 [*]

Epoch: 18/200 - LR: 0.000300
153.3s - loss: 1.402 - acc: 68.750: 100%|████████████████████████| 54000/54000 [02:33<00:00, 309.71it/s]
train loss: 0.546 - train acc: 84.311 - val loss: 0.113 - val acc: 96.317 [*]

Epoch: 19/200 - LR: 0.000300
156.3s - loss: 0.039 - acc: 93.750: 100%|████████████████████████| 54000/54000 [02:36<00:00, 345.51it/s]
train loss: 0.553 - train acc: 84.543 - val loss: 0.188 - val acc: 95.433

Epoch: 20/200 - LR: 0.000300
182.3s - loss: 0.912 - acc: 68.750: 100%|████████████████████████| 54000/54000 [03:02<00:00, 231.90it/s]
train loss: 0.564 - train acc: 84.020 - val loss: 0.213 - val acc: 94.800

Epoch: 21/200 - LR: 0.000300
156.9s - loss: 0.433 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:36<00:00, 354.48it/s]
train loss: 0.589 - train acc: 83.404 - val loss: 0.145 - val acc: 94.850

Epoch: 22/200 - LR: 0.000300
171.1s - loss: 0.564 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:51<00:00, 243.82it/s]
train loss: 0.590 - train acc: 83.189 - val loss: 0.168 - val acc: 95.500

Epoch: 23/200 - LR: 0.000300
184.0s - loss: -0.073 - acc: 93.750: 100%|███████████████████████| 54000/54000 [03:04<00:00, 293.42it/s]
train loss: 0.620 - train acc: 82.443 - val loss: 0.057 - val acc: 94.850

Epoch: 24/200 - LR: 0.000300
195.0s - loss: 0.498 - acc: 68.750: 100%|████████████████████████| 54000/54000 [03:14<00:00, 215.77it/s]
train loss: 0.627 - train acc: 82.209 - val loss: 0.121 - val acc: 94.933

Epoch: 25/200 - LR: 0.000300
157.3s - loss: 0.568 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:37<00:00, 281.71it/s]
train loss: 0.618 - train acc: 82.150 - val loss: 0.133 - val acc: 95.017

Epoch: 26/200 - LR: 0.000300
150.3s - loss: -0.639 - acc: 100.000: 100%|██████████████████████| 54000/54000 [02:30<00:00, 291.28it/s]
train loss: 0.613 - train acc: 81.933 - val loss: 0.168 - val acc: 94.017

Epoch: 27/200 - LR: 0.000300
163.9s - loss: 0.304 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:43<00:00, 329.53it/s]
train loss: 0.627 - train acc: 81.819 - val loss: 0.144 - val acc: 95.933

Epoch: 28/200 - LR: 0.000300
153.0s - loss: 0.380 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:32<00:00, 306.57it/s]
train loss: 0.603 - train acc: 82.076 - val loss: 0.057 - val acc: 95.633

Epoch: 29/200 - LR: 0.000300
172.2s - loss: -0.071 - acc: 93.750: 100%|███████████████████████| 54000/54000 [02:52<00:00, 313.52it/s]
train loss: 0.623 - train acc: 82.124 - val loss: 0.114 - val acc: 96.017

Epoch: 30/200 - LR: 0.000300
143.8s - loss: 0.675 - acc: 81.250: 100%|████████████████████████| 54000/54000 [02:23<00:00, 329.06it/s]
train loss: 0.636 - train acc: 81.933 - val loss: 0.185 - val acc: 95.717

Epoch: 31/200 - LR: 0.000300
166.7s - loss: 1.192 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:46<00:00, 307.40it/s]
train loss: 0.611 - train acc: 82.126 - val loss: 0.173 - val acc: 96.133

Epoch: 32/200 - LR: 0.000300
153.9s - loss: 1.137 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:33<00:00, 350.93it/s]
train loss: 0.581 - train acc: 82.957 - val loss: 0.143 - val acc: 95.783

Epoch: 33/200 - LR: 0.000300
191.4s - loss: 0.138 - acc: 93.750: 100%|████████████████████████| 54000/54000 [03:11<00:00, 282.08it/s]
train loss: 0.593 - train acc: 82.683 - val loss: 0.259 - val acc: 95.650

Epoch: 34/200 - LR: 0.000300
150.6s - loss: 0.535 - acc: 81.250: 100%|████████████████████████| 54000/54000 [02:30<00:00, 363.96it/s]
train loss: 0.642 - train acc: 81.769 - val loss: 0.246 - val acc: 96.000

Epoch: 35/200 - LR: 0.000300
�106.5s - loss: 0.685 - acc: 78.125:  63%|███████████████▏        | 34048/54000 [01:46<00:58, 342.94it/s
171.6s - loss: 0.586 - acc: 81.250: 100%|████████████████████████| 54000/54000 [02:51<00:00, 259.16it/s]
train loss: 0.621 - train acc: 82.106 - val loss: 0.211 - val acc: 95.900

Epoch: 36/200 - LR: 0.000300
174.5s - loss: 0.722 - acc: 75.000: 100%|████████████████████████| 54000/54000 [02:54<00:00, 309.45it/s]
train loss: 0.615 - train acc: 82.026 - val loss: 0.167 - val acc: 96.000

Epoch: 37/200 - LR: 0.000300
168.1s - loss: 0.512 - acc: 81.250: 100%|████████████████████████| 54000/54000 [02:48<00:00, 321.16it/s]
train loss: 0.608 - train acc: 82.265 - val loss: 0.152 - val acc: 96.317

Epoch: 38/200 - LR: 0.000300
155.8s - loss: 0.390 - acc: 87.500: 100%|████████████████████████| 54000/54000 [02:35<00:00, 305.99it/s]
train loss: 0.626 - train acc: 81.854 - val loss: 0.173 - val acc: 96.550 [*]

Epoch: 39/200 - LR: 0.000300
154.8s - loss: 0.108 - acc: 93.750: 100%|████████████████████████| 54000/54000 [02:34<00:00, 348.91it/s]
train loss: 0.634 - train acc: 81.515 - val loss: 0.220 - val acc: 96.183

Epoch: 40/200 - LR: 0.000300
159.1s - loss: -0.091 - acc: 100.000: 100%|██████████████████████| 54000/54000 [02:39<00:00, 339.46it/s]
train loss: 0.618 - train acc: 81.963 - val loss: 0.243 - val acc: 95.600
ifgovh commented 5 years ago

@ipod825 I found the best model in the repo has val acc = 97. So I guess it is normal.

yuby14 commented 4 years ago

I met the same problem. As I run python main.py --is_train 1, the train acc converged to 91% and the val acc converged to 97% about 30 epochs later. But after 30 epoched, the train acc began to decrease and the loss began to increase. Could you tell me what is the problem, please? @ipod825@ifgovh

yuby14 commented 4 years ago

So, I can't understand:1)why the val acc is always higher than the train acc;2)why the train acc began to decrease and the loss began to increase after 30 epoches

ifgovh commented 4 years ago

I guess it is due to the stochasticity caused by reinforcement learning. Maybe the hybrid loss function does not converge well.

yuby14 commented 4 years ago

I have run python main.py --is_train 1 for several times,and the results are the same. What's more,he train acc converged to 91% and I haven't catch higher train acc.