search

locuslab / TCN

Sequence modeling benchmarks and temporal convolutional networks
https://github.com/locuslab/TCN
MIT License
4.18k stars 881 forks source link

Results for adding problem not reproducible with default values #30

Closed sadegh-erc closed 5 years ago

sadegh-erc commented 5 years ago

Hey guys,

Thanks very much for sharing the code for the paper. I tried running the add_test.py with no change in the default values and the loss does not converge. However for some other random seeds it'll converge.

OS: Ubuntu 16.04 PyTorch version: 1.0.0

Output:

Namespace(batch_size=32, clip=-1, cuda=True, dropout=0.0, epochs=10, ksize=7, levels=8, log_interval=100, lr=0.004, nhid=30, optim='Adam', seed=1111, seq_len=400)
Producing data...
Train Epoch:  1 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.531405
Train Epoch:  1 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.166126
Train Epoch:  1 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.171005
Train Epoch:  1 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.170864
Train Epoch:  1 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.166217
Train Epoch:  1 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.172447
Train Epoch:  1 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.166411
Train Epoch:  1 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167221
Train Epoch:  1 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.166980
Train Epoch:  1 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.170149
Train Epoch:  1 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.167781
Train Epoch:  1 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.173033
Train Epoch:  1 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.167806
Train Epoch:  1 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.176322
Train Epoch:  1 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.174221

Test set: Average loss: 0.162485

Train Epoch:  2 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.164098
Train Epoch:  2 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.165515
Train Epoch:  2 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.169491
Train Epoch:  2 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.170406
Train Epoch:  2 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.164345
Train Epoch:  2 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.171381
Train Epoch:  2 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.165580
Train Epoch:  2 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167373
Train Epoch:  2 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.165166
Train Epoch:  2 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.169122
Train Epoch:  2 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.167244
Train Epoch:  2 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.172299
Train Epoch:  2 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.166954
Train Epoch:  2 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.175234
Train Epoch:  2 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.174419

Test set: Average loss: 0.159353

Train Epoch:  3 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.162857
Train Epoch:  3 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.165053
Train Epoch:  3 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168938
Train Epoch:  3 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.170395
Train Epoch:  3 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.163797
Train Epoch:  3 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.171040
Train Epoch:  3 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164996
Train Epoch:  3 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167644
Train Epoch:  3 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164682
Train Epoch:  3 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168531
Train Epoch:  3 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.167023
Train Epoch:  3 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.172070
Train Epoch:  3 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.165484
Train Epoch:  3 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.174374
Train Epoch:  3 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.174143

Test set: Average loss: 0.159425

Train Epoch:  4 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.162641
Train Epoch:  4 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164765
Train Epoch:  4 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168638
Train Epoch:  4 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.170502
Train Epoch:  4 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.163475
Train Epoch:  4 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.170376
Train Epoch:  4 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164857
Train Epoch:  4 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167976
Train Epoch:  4 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164582
Train Epoch:  4 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168683
Train Epoch:  4 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166684
Train Epoch:  4 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.171560
Train Epoch:  4 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.165219
Train Epoch:  4 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.174026
Train Epoch:  4 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.173639

Test set: Average loss: 0.159309

Train Epoch:  5 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.162744
Train Epoch:  5 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164428
Train Epoch:  5 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168446
Train Epoch:  5 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.170423
Train Epoch:  5 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.163086
Train Epoch:  5 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.170119
Train Epoch:  5 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164834
Train Epoch:  5 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167641
Train Epoch:  5 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164558
Train Epoch:  5 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168754
Train Epoch:  5 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166589
Train Epoch:  5 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.171319
Train Epoch:  5 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.165145
Train Epoch:  5 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.173861
Train Epoch:  5 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.173357

Test set: Average loss: 0.159252

Train Epoch:  6 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.163019
Train Epoch:  6 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164251
Train Epoch:  6 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168384
Train Epoch:  6 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.170225
Train Epoch:  6 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.162895
Train Epoch:  6 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.170042
Train Epoch:  6 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164821
Train Epoch:  6 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167372
Train Epoch:  6 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164525
Train Epoch:  6 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168579
Train Epoch:  6 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166480
Train Epoch:  6 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.171241
Train Epoch:  6 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.165065
Train Epoch:  6 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.173829
Train Epoch:  6 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.173157

Test set: Average loss: 0.159345

Train Epoch:  7 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.163180
Train Epoch:  7 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164179
Train Epoch:  7 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168342
Train Epoch:  7 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.169995
Train Epoch:  7 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.162740
Train Epoch:  7 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.169979
Train Epoch:  7 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164783
Train Epoch:  7 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.167070
Train Epoch:  7 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164500
Train Epoch:  7 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168542
Train Epoch:  7 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166456
Train Epoch:  7 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.171119
Train Epoch:  7 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.164998
Train Epoch:  7 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.173744
Train Epoch:  7 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.172998

Test set: Average loss: 0.159525

Train Epoch:  8 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.163299
Train Epoch:  8 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164069
Train Epoch:  8 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168318
Train Epoch:  8 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.169810
Train Epoch:  8 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.162647
Train Epoch:  8 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.169944
Train Epoch:  8 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164746
Train Epoch:  8 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.166869
Train Epoch:  8 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164391
Train Epoch:  8 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168394
Train Epoch:  8 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166326
Train Epoch:  8 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.171027
Train Epoch:  8 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.164934
Train Epoch:  8 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.173725
Train Epoch:  8 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.172872

Test set: Average loss: 0.159705

Train Epoch:  9 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.163319
Train Epoch:  9 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164034
Train Epoch:  9 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168286
Train Epoch:  9 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.169652
Train Epoch:  9 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.162581
Train Epoch:  9 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.169919
Train Epoch:  9 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164722
Train Epoch:  9 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.166746
Train Epoch:  9 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164353
Train Epoch:  9 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168298
Train Epoch:  9 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166287
Train Epoch:  9 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.170995
Train Epoch:  9 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.164889
Train Epoch:  9 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.173729
Train Epoch:  9 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.172800

Test set: Average loss: 0.159804

Train Epoch: 10 [  3168/ 50000 (6%)]    Learning rate: 0.0040   Loss: 0.163326
Train Epoch: 10 [  6368/ 50000 (13%)]   Learning rate: 0.0040   Loss: 0.164024
Train Epoch: 10 [  9568/ 50000 (19%)]   Learning rate: 0.0040   Loss: 0.168285
Train Epoch: 10 [ 12768/ 50000 (26%)]   Learning rate: 0.0040   Loss: 0.169597
Train Epoch: 10 [ 15968/ 50000 (32%)]   Learning rate: 0.0040   Loss: 0.162569
Train Epoch: 10 [ 19168/ 50000 (38%)]   Learning rate: 0.0040   Loss: 0.169920
Train Epoch: 10 [ 22368/ 50000 (45%)]   Learning rate: 0.0040   Loss: 0.164698
Train Epoch: 10 [ 25568/ 50000 (51%)]   Learning rate: 0.0040   Loss: 0.166688
Train Epoch: 10 [ 28768/ 50000 (58%)]   Learning rate: 0.0040   Loss: 0.164336
Train Epoch: 10 [ 31968/ 50000 (64%)]   Learning rate: 0.0040   Loss: 0.168244
Train Epoch: 10 [ 35168/ 50000 (70%)]   Learning rate: 0.0040   Loss: 0.166240
Train Epoch: 10 [ 38368/ 50000 (77%)]   Learning rate: 0.0040   Loss: 0.170987
Train Epoch: 10 [ 41568/ 50000 (83%)]   Learning rate: 0.0040   Loss: 0.164852
Train Epoch: 10 [ 44768/ 50000 (90%)]   Learning rate: 0.0040   Loss: 0.173748
Train Epoch: 10 [ 47968/ 50000 (96%)]   Learning rate: 0.0040   Loss: 0.172743

Test set: Average loss: 0.159894
jerrybai1995 commented 5 years ago

Hi @sadegh-erc!

I just tried to run it and it seems to work well on my end. I would suggest tuning the hyperparameters a bit (shouldn't need too many changes), by starting with sequences with smaller T (e.g. T=100 or so). Then train a model with larger T's by adding larger dilations and more layers.