Open ruizewang opened 4 years ago
Yes, this is a common failure mode! The model also takes a long time to get better-than-chance performance, which can make it look like it's stuck.
Yes, this is a common failure mode! The model also takes a long time to get better-than-chance performance, which can make it look like it's stuck.
- What batch size are you using? Are you training on AudioSet? Note that I trained that model with 3 GPUs, so the effective batch size was 45.
- The loss values that you should probably be looking at are "loss:label", which is the cross-entropy loss, and "acc" which is the overall accuracy. Here, chance performance would be acc = 0.5 and loss:label = ln(0.5) = 0.693. So, it looks like the model has not yet reached chance performance.
- In my experiments, the model took something like 2K iterations to reach chance performance (loss:label = 0.693), and 11K iterations to do better than chance (loss:label = 0.692). So, for a long time it looked like the model was stuck at chance.
- Did you decrease the learning rate? I trained with lr = 1e-2 at the beginning. This might explain why your model is still doing worse than chance at 5K iterations.
Thank you very much for your explanation. This makes me suddenly understand.
Iteration 15080, lr = 1e-02, total:loss: 1.257 reg: 0.068 loss:label: 0.693 acc:label: 0.496, time: 7.318
Iteration 15090, lr = 1e-02, total:loss: 1.257 reg: 0.068 loss:label: 0.693 acc:label: 0.496, time: 7.203
[grad norm:][0.0545679964]
Iteration 15100, lr = 1e-02, total:loss: 1.256 reg: 0.068 loss:label: 0.693 acc:label: 0.495, time: 7.167
Iteration 15110, lr = 1e-02, total:loss: 1.255 reg: 0.068 loss:label: 0.693 acc:label: 0.495, time: 7.123
Iteration 15120, lr = 1e-02, total:loss: 1.257 reg: 0.068 loss:label: 0.693 acc:label: 0.497, time: 7.079
Iteration 15130, lr = 1e-02, total:loss: 1.259 reg: 0.068 loss:label: 0.693 acc:label: 0.498, time: 7.032
Iteration 15140, lr = 1e-02, total:loss: 1.260 reg: 0.068 loss:label: 0.693 acc:label: 0.499, time: 6.990
Iteration 15150, lr = 1e-02, total:loss: 1.259 reg: 0.068 loss:label: 0.693 acc:label: 0.498, time: 7.013
Thanks a lot, Andrew. It is really helpful. š
Sorry to bother you, I am here again. When a shift model (e.g., 'net.tf-30000') has been trained, how to use this model for testing? Only set "is_training" as False, and run shift_net.train? But I think maybe there is something else I should do.
class Model:
def __init__(self, pr, sess, gpus, is_training=False, pr_test=None):
Hello Andrew.
[grad norm:][4.99999952]
Iteration 650000, lr = 1e-02, total:loss: 1.339 reg: 0.047 loss:label: 0.692 acc:label: 0.600, time: 42.686
Iteration 650010, lr = 1e-02, total:loss: 1.339 reg: 0.047 loss:label: 0.700 acc:label: 0.592, time: 38.813
Iteration 650020, lr = 1e-02, total:loss: 1.339 reg: 0.047 loss:label: 0.710 acc:label: 0.582, time: 35.310
Iteration 650030, lr = 1e-02, total:loss: 1.332 reg: 0.047 loss:label: 0.712 acc:label: 0.574, time: 32.127
Iteration 650040, lr = 1e-02, total:loss: 1.328 reg: 0.047 loss:label: 0.711 acc:label: 0.570, time: 29.258
Iteration 650050, lr = 1e-02, total:loss: 1.327 reg: 0.047 loss:label: 0.713 acc:label: 0.568, time: 26.660
Iteration 650060, lr = 1e-02, total:loss: 1.321 reg: 0.047 loss:label: 0.714 acc:label: 0.560, time: 24.311
Iteration 650070, lr = 1e-02, total:loss: 1.316 reg: 0.047 loss:label: 0.713 acc:label: 0.556, time: 22.189
Iteration 650080, lr = 1e-02, total:loss: 1.307 reg: 0.047 loss:label: 0.715 acc:label: 0.545, time: 20.270
Iteration 650090, lr = 1e-02, total:loss: 1.303 reg: 0.047 loss:label: 0.714 acc:label: 0.542, time: 18.539
......
Iteration 652000, lr = 1e-02, total:loss: 1.238 reg: 0.047 loss:label: 0.694 acc:label: 0.498, time: 1.845
Iteration 652010, lr = 1e-02, total:loss: 1.239 reg: 0.047 loss:label: 0.694 acc:label: 0.499, time: 1.855
Iteration 652020, lr = 1e-02, total:loss: 1.240 reg: 0.047 loss:label: 0.694 acc:label: 0.500, time: 1.853
Iteration 652030, lr = 1e-02, total:loss: 1.243 reg: 0.047 loss:label: 0.694 acc:label: 0.503, time: 1.857
Iteration 652040, lr = 1e-02, total:loss: 1.245 reg: 0.047 loss:label: 0.694 acc:label: 0.504, time: 1.857
Iteration 652050, lr = 1e-02, total:loss: 1.244 reg: 0.047 loss:label: 0.694 acc:label: 0.504, time: 1.857
Iteration 652060, lr = 1e-02, total:loss: 1.246 reg: 0.047 loss:label: 0.693 acc:label: 0.505, time: 1.856
Iteration 652070, lr = 1e-02, total:loss: 1.245 reg: 0.047 loss:label: 0.693 acc:label: 0.505, time: 1.859
Iteration 652080, lr = 1e-02, total:loss: 1.245 reg: 0.047 loss:label: 0.694 acc:label: 0.504, time: 1.861
Iteration 652090, lr = 1e-02, total:loss: 1.243 reg: 0.047 loss:label: 0.694 acc:label: 0.503, time: 1.859
[grad norm:][0.266301781]
Iteration 652100, lr = 1e-02, total:loss: 1.242 reg: 0.047 loss:label: 0.694 acc:label: 0.502, time: 1.857
Iteration 652110, lr = 1e-02, total:loss: 1.242 reg: 0.047 loss:label: 0.694 acc:label: 0.501, time: 1.862
Iteration 652120, lr = 1e-02, total:loss: 1.244 reg: 0.047 loss:label: 0.694 acc:label: 0.503, time: 1.861
Iteration 652130, lr = 1e-02, total:loss: 1.241 reg: 0.047 loss:label: 0.694 acc:label: 0.501, time: 1.860
Iteration 652140, lr = 1e-02, total:loss: 1.241 reg: 0.047 loss:label: 0.694 acc:label: 0.501, time: 1.856
Iteration 652150, lr = 1e-02, total:loss: 1.242 reg: 0.047 loss:label: 0.694 acc:label: 0.502, time: 1.854
Iteration 652160, lr = 1e-02, total:loss: 1.242 reg: 0.047 loss:label: 0.694 acc:label: 0.502, time: 1.850
Iteration 652170, lr = 1e-02, total:loss: 1.243 reg: 0.047 loss:label: 0.694 acc:label: 0.502, time: 1.848
Iteration 652180, lr = 1e-02, total:loss: 1.244 reg: 0.047 loss:label: 0.694 acc:label: 0.504, time: 1.846
Iteration 652190, lr = 1e-02, total:loss: 1.243 reg: 0.047 loss:label: 0.693 acc:label: 0.503, time: 1.850
[grad norm:][0.122649804]
Iteration 652200, lr = 1e-02, total:loss: 1.246 reg: 0.047 loss:label: 0.694 acc:label: 0.506, time: 1.848
Iteration 652210, lr = 1e-02, total:loss: 1.244 reg: 0.047 loss:label: 0.694 acc:label: 0.504, time: 1.846
Iteration 652220, lr = 1e-02, total:loss: 1.247 reg: 0.047 loss:label: 0.694 acc:label: 0.507, time: 1.847
Iteration 652230, lr = 1e-02, total:loss: 1.246 reg: 0.047 loss:label: 0.693 acc:label: 0.506, time: 1.849
Iteration 652240, lr = 1e-02, total:loss: 1.246 reg: 0.047 loss:label: 0.694 acc:label: 0.506, time: 1.852
Iteration 652250, lr = 1e-02, total:loss: 1.246 reg: 0.047 loss:label: 0.694 acc:label: 0.506, time: 1.850
Iteration 652260, lr = 1e-02, total:loss: 1.245 reg: 0.047 loss:label: 0.694 acc:label: 0.505, time: 1.852
Iteration 652270, lr = 1e-02, total:loss: 1.245 reg: 0.047 loss:label: 0.693 acc:label: 0.505, time: 1.850
Iteration 652280, lr = 1e-02, total:loss: 1.248 reg: 0.047 loss:label: 0.693 acc:label: 0.508, time: 1.849
Iteration 652290, lr = 1e-02, total:loss: 1.244 reg: 0.047 loss:label: 0.693 acc:label: 0.504, time: 1.846
[grad norm:][0.0564598292]
Iteration 652300, lr = 1e-02, total:loss: 1.246 reg: 0.047 loss:label: 0.693 acc:label: 0.506, time: 1.845
shift_example.py
for an example of testing a trained network.
- Please refer to
shift_example.py
for an example of testing a trained network.- I think the loss is going up when you fine-tune because you are using a higher learning rate and (especially) a smaller batch size. The model starts out better than chance, but the parameters become worse because it's taking large steps (high learning rate) in not-so-great gradient directions (low batch size).
shift_example.py
. But I saw you reported accuracy in the paper ---"Task performance. We found that the model obtained 59.9% accuracy on held-out videos for its alignment task (chance = 50%)." Actually, I want to test the model and get the accuracy result on the test dataset. Do I need to re-write this part code?There is an example of generating a cam in shift_example.py. But I saw you reported accuracy in the paper ---"Task performance. We found that the model obtained 59.9% accuracy on held-out videos for its alignment task (chance = 50%)." Actually, I want to test the model and get the accuracy result on the test dataset. Do I need to re-write this part code?
This problem is solved. As your suggestion, I add a "test_accuracy" function in "class NetClf". Thanks again, Andrew. š
Hi @ruizewang, would you mind to share the code you use to create data file for training, I would really appreciate that
Hello, When I train shift_lowfps model, the loss decreases slowly, but acc doesn't change (0.500). Could you give me some advice?
[grad norm:][0.0125109516] Iteration 5500, lr = 1e-03, total:loss: 1.246 reg: 0.041 loss:label: 0.705 acc:label: 0.500, time: 2.978 Iteration 5510, lr = 1e-03, total:loss: 1.244 reg: 0.040 loss:label: 0.704 acc:label: 0.500, time: 2.974 Iteration 5520, lr = 1e-03, total:loss: 1.241 reg: 0.039 loss:label: 0.703 acc:label: 0.500, time: 2.953 Iteration 5530, lr = 1e-03, total:loss: 1.239 reg: 0.037 loss:label: 0.702 acc:label: 0.500, time: 2.960 Iteration 5540, lr = 1e-03, total:loss: 1.238 reg: 0.036 loss:label: 0.701 acc:label: 0.500, time: 2.971 Iteration 5550, lr = 1e-03, total:loss: 1.236 reg: 0.035 loss:label: 0.700 acc:label: 0.500, time: 2.965 Iteration 5560, lr = 1e-03, total:loss: 1.234 reg: 0.034 loss:label: 0.700 acc:label: 0.500, time: 2.961 Iteration 5570, lr = 1e-03, total:loss: 1.232 reg: 0.033 loss:label: 0.699 acc:label: 0.500, time: 2.957 Iteration 5580, lr = 1e-03, total:loss: 1.231 reg: 0.032 loss:label: 0.699 acc:label: 0.500, time: 2.952 Iteration 5590, lr = 1e-03, total:loss: 1.229 reg: 0.031 loss:label: 0.698 acc:label: 0.500, time: 2.967 [grad norm:][0.00501754601] Iteration 5600, lr = 1e-03, total:loss: 1.228 reg: 0.030 loss:label: 0.698 acc:label: 0.500, time: 2.968 Iteration 5610, lr = 1e-03, total:loss: 1.227 reg: 0.030 loss:label: 0.697 acc:label: 0.500, time: 2.960 Iteration 5620, lr = 1e-03, total:loss: 1.225 reg: 0.029 loss:label: 0.697 acc:label: 0.500, time: 2.951 Iteration 5630, lr = 1e-03, total:loss: 1.224 reg: 0.028 loss:label: 0.696 acc:label: 0.500, time: 2.977 Iteration 5640, lr = 1e-03, total:loss: 1.223 reg: 0.027 loss:label: 0.696 acc:label: 0.500, time: 2.973 Iteration 5650, lr = 1e-03, total:loss: 1.222 reg: 0.026 loss:label: 0.696 acc:label: 0.500, time: 2.981