search

fychao / gpu_benchmark_tools

this is collection of gpu benchmark tools
GNU General Public License v3.0
1 stars 1 forks source link

OpenNMT test #7

Open fychao opened 6 years ago

fychao commented 6 years ago

on test data

[01/24/18 09:54:24 INFO] Epoch 13 ; Iteration 50/92 ; Optim SGD LR 0.240100 ; Source tokens/s 7603 ; Perplexity 262.84
[01/24/18 09:54:38 INFO] Epoch 13 ; Iteration 92/92 ; Optim SGD LR 0.240100 ; Source tokens/s 7492 ; Perplexity 270.82
[01/24/18 09:54:38 INFO] Evaluating on the validation dataset...
[01/24/18 09:54:42 INFO] Validation perplexity: 804.17
[01/24/18 09:54:42 INFO] Saving checkpoint to 'demo_epoch13_804.17.t7'...

real    8m27.912s
user    10m41.568s
sys     2m6.540s
fychao commented 6 years ago 
[01/25/18 08:56:32 INFO] Epoch 2 ; Iteration 32250/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7265 ; Perplexity 12.73
[01/25/18 08:56:52 INFO] Epoch 2 ; Iteration 32300/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7168 ; Perplexity 12.78
[01/25/18 08:57:12 INFO] Epoch 2 ; Iteration 32350/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7209 ; Perplexity 12.81
[01/25/18 08:57:27 INFO] Epoch 2 ; Iteration 32389/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7062 ; Perplexity 12.47
[01/25/18 08:57:27 INFO] Evaluating on the validation dataset...
[01/25/18 08:57:32 INFO] Validation perplexity: 10.00
[01/25/18 08:57:32 INFO] Saving checkpoint to '../wmt15-de-en/wmt15-all-en-de_epoch2_10.00.t7'...
[01/25/18 08:57:33 INFO]
[01/25/18 08:58:01 INFO] Epoch 3 ; Iteration 50/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7603 ; Perplexity 12.51
[01/25/18 08:58:21 INFO] Epoch 3 ; Iteration 100/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7971 ; Perplexity 12.52
[01/25/18 08:58:41 INFO] Epoch 3 ; Iteration 150/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7420 ; Perplexity 12.12
[01/25/18 08:59:02 INFO] Epoch 3 ; Iteration 200/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7878 ; Perplexity 12.31
[01/25/18 08:59:22 INFO] Epoch 3 ; Iteration 250/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7152 ; Perplexity 12.59
[01/25/18 08:59:42 INFO] Epoch 3 ; Iteration 300/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7353 ; Perplexity 12.39
[01/25/18 09:00:02 INFO] Epoch 3 ; Iteration 350/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7124 ; Perplexity 12.60
[01/25/18 09:00:21 INFO] Epoch 3 ; Iteration 400/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7198 ; Perplexity 11.95
[01/25/18 09:00:42 INFO] Epoch 3 ; Iteration 450/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7600 ; Perplexity 12.48
[01/25/18 09:01:02 INFO] Epoch 3 ; Iteration 500/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7716 ; Perplexity 12.28
[01/25/18 09:01:23 INFO] Epoch 3 ; Iteration 550/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7425 ; Perplexity 12.59
[01/25/18 09:01:43 INFO] Epoch 3 ; Iteration 600/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 6802 ; Perplexity 12.57
[01/25/18 09:02:02 INFO] Epoch 3 ; Iteration 650/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7617 ; Perplexity 11.99
[01/25/18 09:02:23 INFO] Epoch 3 ; Iteration 700/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7707 ; Perplexity 12.46
[01/25/18 09:02:43 INFO] Epoch 3 ; Iteration 750/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7290 ; Perplexity 12.36
[01/25/18 09:03:03 INFO] Epoch 3 ; Iteration 800/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7364 ; Perplexity 12.20
[01/25/18 09:03:22 INFO] Epoch 3 ; Iteration 850/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7024 ; Perplexity 12.17
[01/25/18 09:03:42 INFO] Epoch 3 ; Iteration 900/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 6932 ; Perplexity 12.04
[01/25/18 09:04:02 INFO] Epoch 3 ; Iteration 950/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7204 ; Perplexity 12.44
[01/25/18 09:04:22 INFO] Epoch 3 ; Iteration 1000/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7013 ; Perplexity 12.04
[01/25/18 09:04:42 INFO] Epoch 3 ; Iteration 1050/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7438 ; Perplexity 12.31
[01/25/18 09:05:02 INFO] Epoch 3 ; Iteration 1100/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7385 ; Perplexity 12.37
[01/25/18 09:05:22 INFO] Epoch 3 ; Iteration 1150/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7860 ; Perplexity 12.20
[01/25/18 09:05:43 INFO] Epoch 3 ; Iteration 1200/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7546 ; Perplexity 12.87
[01/25/18 09:06:02 INFO] Epoch 3 ; Iteration 1250/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7231 ; Perplexity 12.21
[01/25/18 09:06:22 INFO] Epoch 3 ; Iteration 1300/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7217 ; Perplexity 12.58
[01/25/18 09:06:42 INFO] Epoch 3 ; Iteration 1350/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7406 ; Perplexity 12.17
[01/25/18 09:07:02 INFO] Epoch 3 ; Iteration 1400/32389 ; Optim SGD LR 1.000000 ; Source tokens/s 7214 ; Perplexity 12.22
^C/root/torch/install/bin/luajit: ./onmt/data/Batch.lua:136: interrupted!
stack traceback:
        ./onmt/data/Batch.lua:136: in function '__init'
        /root/torch/install/share/lua/5.1/torch/init.lua:91: in function 'getBatch'
        ./onmt/train/Trainer.lua:232: in function 'trainEpoch'
        ./onmt/train/Trainer.lua:439: in function 'train'
        train.lua:337: in function 'main'
        train.lua:342: in main chunk
        [C]: in function 'dofile'
        /root/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk
        [C]: at 0x00405d50

real    448m16.392s
user    594m27.960s
sys     170m3.896s
fychao commented 6 years ago 

#mkdir wmt15-ende
#cd wmt15-ende
#wget "https://s3.amazonaws.com/opennmt-trainingdata/wmt15-de-en.tgz"

git clone https://github.com/OpenNMT/OpenNMT.git
cd OpenNMT
luarocks install tds
for f in ../wmt15-de-en/*.?? ; do th tools/tokenize.lua < $f > $f.tok ; done
for l in en de ; do cat ../wmt15-de-en/commoncrawl.de-en.$l.tok ../wmt15-de-en/europarl-v7.de-en.$l.tok ../wmt15-de-en/news-commentary-v10.de-en.$l.tok > ../wmt15-de-en/wmt15-all-de-en.$l.tok ; done
th preprocess.lua -train_src ../wmt15-de-en/wmt15-all-de-en.en.tok -train_tgt ../wmt15-de-en/wmt15-all-de-en.de.tok -valid_src ../wmt15-de-en/newstest2013.en.tok -valid_tgt ../wmt15-de-en/newstest2013.de.tok -save_data ../wmt15-de-en/wmt15-all-en-de
fychao commented 6 years ago

( 32389 / 50 ) * 20 sec = 32389 sec

2 Epoch reach Perplexity 10.0

fychao commented 6 years ago

root@ca67b9fa31c4:~/wmt15-ende/OpenNMT# time th train.lua -data ../wmt15-de-en/wmt15-all-en-de-train.t7 -save_model ../wmt15-de-en/wmt15-all-en-de -gpuid 1 -end_epoch 1

[01/25/18 14:23:35 INFO] Epoch 1 ; Iteration 64650/64777 ; Optim SGD LR 1.000000 ; Source tokens/s 5324 ; Perplexity 15.20 [01/25/18 14:23:49 INFO] Epoch 1 ; Iteration 64700/64777 ; Optim SGD LR 1.000000 ; Source tokens/s 5269 ; Perplexity 16.12 [01/25/18 14:24:04 INFO] Epoch 1 ; Iteration 64750/64777 ; Optim SGD LR 1.000000 ; Source tokens/s 5205 ; Perplexity 16.21 [01/25/18 14:24:12 INFO] Epoch 1 ; Iteration 64777/64777 ; Optim SGD LR 1.000000 ; Source tokens/s 4918 ; Perplexity 15.47 [01/25/18 14:24:12 INFO] Evaluating on the validation dataset... [01/25/18 14:24:16 INFO] Validation perplexity: 12.57 [01/25/18 14:24:16 INFO] Saving checkpoint to '../wmt15-de-en/wmt15-all-en-de_epoch1_12.57.t7'...

real 306m51.470s user 235m14.920s sys 71m45.520s

fychao commented 6 years ago 
root@ca67b9fa31c4:~# bash go_OpenNMT.sh
Preparing Datasets (tokenizing)...
Tokenization completed in 361.448 seconds - 2399123 sentences
Tokenization completed in 337.380 seconds - 2399123 sentences
Tokenization completed in 335.578 seconds - 1920209 sentences
Tokenization completed in 278.581 seconds - 1920209 sentences
Tokenization completed in 39.957 seconds - 216190 sentences
Tokenization completed in 33.335 seconds - 216190 sentences
Tokenization completed in 0.441 seconds - 3000 sentences
Tokenization completed in 0.389 seconds - 3000 sentences
Done: 1389 seconds elapsed.
Preparing Datasets (preprocessing)...
[01/26/18 02:20:57 INFO] Using on-the-fly 'space' tokenization for input 1
[01/26/18 02:20:57 INFO] Using on-the-fly 'space' tokenization for input 2
[01/26/18 02:20:58 INFO] Preparing vocabulary...
[01/26/18 02:20:58 INFO]  * Building source vocabularies...
[01/26/18 02:23:05 INFO]  * Created word dictionary of size 50004 (pruned from 882957)
[01/26/18 02:23:05 INFO]
[01/26/18 02:23:05 INFO]  * Building target vocabularies...
[01/26/18 02:25:18 INFO]  * Created word dictionary of size 50004 (pruned from 1851345)
[01/26/18 02:25:18 INFO]
[01/26/18 02:25:18 INFO] Preparing training data...
[01/26/18 02:25:18 INFO] --- Preparing train sample
[01/26/18 02:30:33 INFO]  * [-] file '/root/wmt15-ende/wmt15-de-en/wmt15-all-de-en.en.tok' (): 4535522 total, 4535522 drawn, 4144042 kept - unknown words: source = 2.8%, target = 6.1%
[01/26/18 02:30:33 INFO] ... shuffling sentences
[01/26/18 02:31:02 INFO] ... sorting sentences by size
[01/26/18 02:31:27 INFO] Prepared 4144042 sentences:
[01/26/18 02:31:27 INFO]  * 391480 sequences not validated (length, other)
[01/26/18 02:31:27 INFO]  * average sequence length: source = 22.9, target = 21.8
[01/26/18 02:31:27 INFO]  * source sentence length (range of 10): [ 7% ; 32% ; 28% ; 16% ; 8% ; 3% ; 1% ; 0% ; 0% ; 0% ]
[01/26/18 02:31:27 INFO]  * target sentence length (range of 10): [ 8% ; 35% ; 27% ; 15% ; 7% ; 3% ; 1% ; 0% ; 0% ; 0% ]
[01/26/18 02:31:27 INFO]
[01/26/18 02:31:27 INFO]
[01/26/18 02:31:27 INFO] Preparing validation data...
[01/26/18 02:31:27 INFO] --- Preparing valid sample
[01/26/18 02:31:27 INFO]  * [-] file '/root/wmt15-ende/wmt15-de-en/newstest2013.en.tok' (): 3000 total, 3000 drawn, 2891 kept - unknown words: source = 3.3%, target = 6.7%
[01/26/18 02:31:27 INFO] ... shuffling sentences
[01/26/18 02:31:27 INFO] ... sorting sentences by size
[01/26/18 02:31:27 INFO] Prepared 2891 sentences:
[01/26/18 02:31:27 INFO]  * 109 sequences not validated (length, other)
[01/26/18 02:31:27 INFO]  * average sequence length: source = 20.4, target = 19.8
[01/26/18 02:31:27 INFO]  * source sentence length (range of 10): [ 13% ; 36% ; 27% ; 13% ; 5% ; 2% ; 0% ; 0% ; 0% ; 0% ]
[01/26/18 02:31:27 INFO]  * target sentence length (range of 10): [ 15% ; 37% ; 25% ; 13% ; 5% ; 1% ; 0% ; 0% ; 0% ; 0% ]
[01/26/18 02:31:27 INFO]
[01/26/18 02:31:27 INFO]
[01/26/18 02:31:27 INFO] Saving source vocabulary to '/root/wmt15-ende/wmt15-de-en/wmt15-all-en-de.src.dict'...
[01/26/18 02:31:27 INFO] Saving target vocabulary to '/root/wmt15-ende/wmt15-de-en/wmt15-all-en-de.tgt.dict'...
[01/26/18 02:31:27 INFO] Saving data to '/root/wmt15-ende/wmt15-de-en/wmt15-all-en-de-train.t7'...

real    13m54.244s
user    13m47.204s
sys     0m7.964s
Done: 1389 seconds elapsed.
fychao commented 6 years ago 
benchmark01@NGC129:~$ bash go_OpenNMT.sh
Training ...
[02/13/18 03:04:05 INFO] Using GPU(s): 1, 2, 3, 4, 5, 6, 7, 8
[02/13/18 03:04:05 WARNING] For improved efficiency with multiple GPUs, consider installing nccl
[02/13/18 03:04:05 INFO] Training Sequence to Sequence with Attention model...
[02/13/18 03:04:05 INFO] Loading data from '/home/benchmark01/wmt15-ende/wmt15-de-en/wmt15-all-en-de-train.t7'...
[02/13/18 03:06:11 INFO]  * vocabulary size: source = 50004; target = 50004
[02/13/18 03:06:11 INFO]  * additional features: source = 0; target = 0
[02/13/18 03:06:11 INFO]  * maximum sequence length: source = 50; target = 51
[02/13/18 03:06:11 INFO]  * number of training sentences: 4144042
[02/13/18 03:06:11 INFO]  * number of batches: 64777
[02/13/18 03:06:11 INFO]    - source sequence lengths: equal
[02/13/18 03:06:11 INFO]    - maximum size: 64
[02/13/18 03:06:11 INFO]    - average size: 63.97
[02/13/18 03:06:11 INFO]    - capacity: 100.00%
[02/13/18 03:06:11 INFO] Building model...
[02/13/18 03:06:11 INFO]  * Encoder:
[02/13/18 03:06:13 INFO]    - word embeddings size: 500
[02/13/18 03:06:13 INFO]    - type: unidirectional RNN
[02/13/18 03:06:13 INFO]    - structure: cell = LSTM; layers = 2; rnn_size = 500; dropout = 0.3 (naive)
[02/13/18 03:06:13 INFO]  * Decoder:
[02/13/18 03:06:14 INFO]    - word embeddings size: 500
[02/13/18 03:06:14 INFO]    - attention: global (general)
[02/13/18 03:06:14 INFO]    - structure: cell = LSTM; layers = 2; rnn_size = 500; dropout = 0.3 (naive)
[02/13/18 03:06:14 INFO]  * Bridge: copy
[02/13/18 03:10:22 INFO] Initializing parameters...
[02/13/18 03:10:24 INFO]  * number of parameters: 84822004
[02/13/18 03:10:24 INFO] Preparing memory optimization...
[02/13/18 03:10:26 INFO]  * sharing 69% of output/gradInput tensors memory between clones
[02/13/18 03:30:30 INFO] Start training from epoch 1 to 1...
[02/13/18 03:30:30 INFO]
[02/13/18 03:32:37 INFO] Epoch 1 ; Iteration 50/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5470 ; Perplexity 204924.39
[02/13/18 03:34:20 INFO] Epoch 1 ; Iteration 100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5812 ; Perplexity 12039.10
[02/13/18 03:35:55 INFO] Epoch 1 ; Iteration 150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6083 ; Perplexity 3872.96
[02/13/18 03:37:35 INFO] Epoch 1 ; Iteration 200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5878 ; Perplexity 2111.12
[02/13/18 03:39:13 INFO] Epoch 1 ; Iteration 250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5835 ; Perplexity 1536.08
[02/13/18 03:40:49 INFO] Epoch 1 ; Iteration 300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5976 ; Perplexity 1268.20
[02/13/18 03:42:30 INFO] Epoch 1 ; Iteration 350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5952 ; Perplexity 1096.65
[02/13/18 03:44:07 INFO] Epoch 1 ; Iteration 400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5818 ; Perplexity 987.36
[02/13/18 03:45:44 INFO] Epoch 1 ; Iteration 450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5894 ; Perplexity 865.11
[02/13/18 03:47:28 INFO] Epoch 1 ; Iteration 500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5846 ; Perplexity 786.13
[02/13/18 03:49:07 INFO] Epoch 1 ; Iteration 550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5968 ; Perplexity 692.19
[02/13/18 03:50:35 INFO] Epoch 1 ; Iteration 600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6686 ; Perplexity 638.69
[02/13/18 03:52:17 INFO] Epoch 1 ; Iteration 650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5913 ; Perplexity 574.56
[02/13/18 03:53:59 INFO] Epoch 1 ; Iteration 700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5783 ; Perplexity 525.06
[02/13/18 03:55:34 INFO] Epoch 1 ; Iteration 750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6154 ; Perplexity 473.45
[02/13/18 03:57:16 INFO] Epoch 1 ; Iteration 800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5781 ; Perplexity 435.39
[02/13/18 03:58:52 INFO] Epoch 1 ; Iteration 850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6182 ; Perplexity 405.32
[02/13/18 04:00:28 INFO] Epoch 1 ; Iteration 900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5950 ; Perplexity 371.20
[02/13/18 04:02:10 INFO] Epoch 1 ; Iteration 950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5793 ; Perplexity 362.61
[02/13/18 04:03:49 INFO] Epoch 1 ; Iteration 1000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5835 ; Perplexity 335.08
[02/13/18 04:05:31 INFO] Epoch 1 ; Iteration 1050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5669 ; Perplexity 307.62
[02/13/18 04:07:12 INFO] Epoch 1 ; Iteration 1100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5682 ; Perplexity 289.39
[02/13/18 04:08:54 INFO] Epoch 1 ; Iteration 1150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5660 ; Perplexity 285.26
[02/13/18 04:10:36 INFO] Epoch 1 ; Iteration 1200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5597 ; Perplexity 261.54
[02/13/18 04:12:08 INFO] Epoch 1 ; Iteration 1250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6427 ; Perplexity 262.63
[02/13/18 04:13:38 INFO] Epoch 1 ; Iteration 1300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6666 ; Perplexity 247.14
[02/13/18 04:15:17 INFO] Epoch 1 ; Iteration 1350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6005 ; Perplexity 236.21
[02/13/18 04:16:56 INFO] Epoch 1 ; Iteration 1400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5928 ; Perplexity 222.01
[02/13/18 04:18:36 INFO] Epoch 1 ; Iteration 1450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5945 ; Perplexity 207.95
[02/13/18 04:20:17 INFO] Epoch 1 ; Iteration 1500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5803 ; Perplexity 199.36
[02/13/18 04:21:56 INFO] Epoch 1 ; Iteration 1550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5883 ; Perplexity 187.57
[02/13/18 04:23:38 INFO] Epoch 1 ; Iteration 1600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5739 ; Perplexity 178.70
[02/13/18 04:25:14 INFO] Epoch 1 ; Iteration 1650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6224 ; Perplexity 171.50
[02/13/18 04:26:57 INFO] Epoch 1 ; Iteration 1700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5867 ; Perplexity 163.35
[02/13/18 04:28:31 INFO] Epoch 1 ; Iteration 1750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6133 ; Perplexity 148.48
[02/13/18 04:30:05 INFO] Epoch 1 ; Iteration 1800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6308 ; Perplexity 142.88
[02/13/18 04:31:42 INFO] Epoch 1 ; Iteration 1850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5997 ; Perplexity 136.23
[02/13/18 04:33:23 INFO] Epoch 1 ; Iteration 1900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5803 ; Perplexity 127.67
[02/13/18 04:34:50 INFO] Epoch 1 ; Iteration 1950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6699 ; Perplexity 122.66
[02/13/18 04:36:27 INFO] Epoch 1 ; Iteration 2000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6072 ; Perplexity 117.84
[02/13/18 04:38:03 INFO] Epoch 1 ; Iteration 2050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 5988 ; Perplexity 111.09
[02/13/18 04:39:31 INFO] Epoch 1 ; Iteration 2100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6547 ; Perplexity 104.45
[02/13/18 04:41:02 INFO] Epoch 1 ; Iteration 2150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6360 ; Perplexity 99.44
[02/13/18 04:42:37 INFO] Epoch 1 ; Iteration 2200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6590 ; Perplexity 104.11
[02/13/18 04:44:08 INFO] Epoch 1 ; Iteration 2250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6640 ; Perplexity 94.35
[02/13/18 04:45:35 INFO] Epoch 1 ; Iteration 2300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7018 ; Perplexity 89.32
[02/13/18 04:47:02 INFO] Epoch 1 ; Iteration 2350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6543 ; Perplexity 83.03
[02/13/18 04:48:31 INFO] Epoch 1 ; Iteration 2400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6739 ; Perplexity 83.95
[02/13/18 04:49:58 INFO] Epoch 1 ; Iteration 2450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6560 ; Perplexity 80.54
[02/13/18 04:51:26 INFO] Epoch 1 ; Iteration 2500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6519 ; Perplexity 75.90
[02/13/18 04:52:55 INFO] Epoch 1 ; Iteration 2550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6600 ; Perplexity 74.28
[02/13/18 04:54:23 INFO] Epoch 1 ; Iteration 2600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6554 ; Perplexity 70.78
[02/13/18 04:55:53 INFO] Epoch 1 ; Iteration 2650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6565 ; Perplexity 71.44
[02/13/18 04:57:21 INFO] Epoch 1 ; Iteration 2700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6821 ; Perplexity 68.12
[02/13/18 04:58:53 INFO] Epoch 1 ; Iteration 2750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6587 ; Perplexity 66.38
[02/13/18 05:00:24 INFO] Epoch 1 ; Iteration 2800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6410 ; Perplexity 63.58
[02/13/18 05:01:48 INFO] Epoch 1 ; Iteration 2850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6683 ; Perplexity 61.16
[02/13/18 05:03:18 INFO] Epoch 1 ; Iteration 2900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6547 ; Perplexity 61.23
[02/13/18 05:04:42 INFO] Epoch 1 ; Iteration 2950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7095 ; Perplexity 59.12
[02/13/18 05:06:10 INFO] Epoch 1 ; Iteration 3000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6715 ; Perplexity 58.62
[02/13/18 05:07:36 INFO] Epoch 1 ; Iteration 3050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6567 ; Perplexity 54.01
[02/13/18 05:09:06 INFO] Epoch 1 ; Iteration 3100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6674 ; Perplexity 56.70
[02/13/18 05:10:34 INFO] Epoch 1 ; Iteration 3150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6705 ; Perplexity 52.58
[02/13/18 05:12:02 INFO] Epoch 1 ; Iteration 3200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6660 ; Perplexity 52.58
[02/13/18 05:13:29 INFO] Epoch 1 ; Iteration 3250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6896 ; Perplexity 50.96
[02/13/18 05:14:58 INFO] Epoch 1 ; Iteration 3300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6587 ; Perplexity 49.37
[02/13/18 05:16:26 INFO] Epoch 1 ; Iteration 3350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6494 ; Perplexity 49.35
[02/13/18 05:17:54 INFO] Epoch 1 ; Iteration 3400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6737 ; Perplexity 48.13
[02/13/18 05:19:24 INFO] Epoch 1 ; Iteration 3450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6494 ; Perplexity 46.87
[02/13/18 05:20:49 INFO] Epoch 1 ; Iteration 3500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6676 ; Perplexity 45.63
[02/13/18 05:22:16 INFO] Epoch 1 ; Iteration 3550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6910 ; Perplexity 45.90
[02/13/18 05:23:47 INFO] Epoch 1 ; Iteration 3600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6811 ; Perplexity 45.42
[02/13/18 05:25:15 INFO] Epoch 1 ; Iteration 3650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6510 ; Perplexity 45.58
[02/13/18 05:26:43 INFO] Epoch 1 ; Iteration 3700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6681 ; Perplexity 43.66
[02/13/18 05:28:11 INFO] Epoch 1 ; Iteration 3750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6533 ; Perplexity 41.95
[02/13/18 05:29:32 INFO] Epoch 1 ; Iteration 3800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7121 ; Perplexity 41.26
[02/13/18 05:30:55 INFO] Epoch 1 ; Iteration 3850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7286 ; Perplexity 42.27
[02/13/18 05:32:11 INFO] Epoch 1 ; Iteration 3900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7641 ; Perplexity 40.76
[02/13/18 05:33:37 INFO] Epoch 1 ; Iteration 3950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7097 ; Perplexity 39.87
[02/13/18 05:34:57 INFO] Epoch 1 ; Iteration 4000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7174 ; Perplexity 38.54
[02/13/18 05:36:19 INFO] Epoch 1 ; Iteration 4050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7079 ; Perplexity 38.88
[02/13/18 05:37:40 INFO] Epoch 1 ; Iteration 4100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7327 ; Perplexity 39.01
[02/13/18 05:39:02 INFO] Epoch 1 ; Iteration 4150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7377 ; Perplexity 39.99
[02/13/18 05:40:24 INFO] Epoch 1 ; Iteration 4200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6881 ; Perplexity 36.52
[02/13/18 05:41:45 INFO] Epoch 1 ; Iteration 4250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7253 ; Perplexity 36.21
[02/13/18 05:43:02 INFO] Epoch 1 ; Iteration 4300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7761 ; Perplexity 36.57
[02/13/18 05:44:25 INFO] Epoch 1 ; Iteration 4350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6717 ; Perplexity 35.41
[02/13/18 05:45:47 INFO] Epoch 1 ; Iteration 4400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7010 ; Perplexity 34.74
[02/13/18 05:47:07 INFO] Epoch 1 ; Iteration 4450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7410 ; Perplexity 35.48
[02/13/18 05:48:30 INFO] Epoch 1 ; Iteration 4500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 6827 ; Perplexity 34.06
[02/13/18 05:49:49 INFO] Epoch 1 ; Iteration 4550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7279 ; Perplexity 33.22
[02/13/18 05:51:10 INFO] Epoch 1 ; Iteration 4600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7341 ; Perplexity 34.24
[02/13/18 05:52:33 INFO] Epoch 1 ; Iteration 4650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7160 ; Perplexity 33.01
[02/13/18 05:53:52 INFO] Epoch 1 ; Iteration 4700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7458 ; Perplexity 33.42
[02/13/18 05:55:14 INFO] Epoch 1 ; Iteration 4750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7060 ; Perplexity 33.14
[02/13/18 05:56:36 INFO] Epoch 1 ; Iteration 4800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7104 ; Perplexity 32.97
[02/13/18 05:57:56 INFO] Epoch 1 ; Iteration 4850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7468 ; Perplexity 32.04
[02/13/18 05:59:13 INFO] Epoch 1 ; Iteration 4900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7438 ; Perplexity 31.02
[02/13/18 06:00:34 INFO] Epoch 1 ; Iteration 4950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7220 ; Perplexity 31.40
[02/13/18 06:01:52 INFO] Epoch 1 ; Iteration 5000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7323 ; Perplexity 31.41
[02/13/18 06:01:52 INFO] Saving checkpoint to '/home/benchmark01/wmt15-ende/wmt15-de-en/wmt15-all-en-de_checkpoint.t7'...
[02/13/18 06:03:13 INFO] Epoch 1 ; Iteration 5050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7207 ; Perplexity 30.71
[02/13/18 06:04:33 INFO] Epoch 1 ; Iteration 5100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7415 ; Perplexity 30.54
[02/13/18 06:05:53 INFO] Epoch 1 ; Iteration 5150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7113 ; Perplexity 29.57
[02/13/18 06:07:10 INFO] Epoch 1 ; Iteration 5200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7562 ; Perplexity 29.99
[02/13/18 06:08:28 INFO] Epoch 1 ; Iteration 5250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7510 ; Perplexity 29.48
[02/13/18 06:09:48 INFO] Epoch 1 ; Iteration 5300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7382 ; Perplexity 29.86
[02/13/18 06:11:06 INFO] Epoch 1 ; Iteration 5350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7500 ; Perplexity 29.40
[02/13/18 06:12:24 INFO] Epoch 1 ; Iteration 5400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7233 ; Perplexity 28.39
[02/13/18 06:13:37 INFO] Epoch 1 ; Iteration 5450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7912 ; Perplexity 28.49
[02/13/18 06:14:53 INFO] Epoch 1 ; Iteration 5500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7714 ; Perplexity 28.17
[02/13/18 06:16:11 INFO] Epoch 1 ; Iteration 5550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7436 ; Perplexity 28.26
[02/13/18 06:17:27 INFO] Epoch 1 ; Iteration 5600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7468 ; Perplexity 27.11
[02/13/18 06:18:46 INFO] Epoch 1 ; Iteration 5650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7727 ; Perplexity 28.11
[02/13/18 06:20:03 INFO] Epoch 1 ; Iteration 5700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7672 ; Perplexity 27.98
[02/13/18 06:21:19 INFO] Epoch 1 ; Iteration 5750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7726 ; Perplexity 26.75
[02/13/18 06:22:36 INFO] Epoch 1 ; Iteration 5800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7571 ; Perplexity 28.63
[02/13/18 06:23:52 INFO] Epoch 1 ; Iteration 5850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7692 ; Perplexity 27.09
[02/13/18 06:25:08 INFO] Epoch 1 ; Iteration 5900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7651 ; Perplexity 26.57
[02/13/18 06:26:25 INFO] Epoch 1 ; Iteration 5950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7740 ; Perplexity 26.47
[02/13/18 06:27:41 INFO] Epoch 1 ; Iteration 6000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7687 ; Perplexity 26.41
[02/13/18 06:28:59 INFO] Epoch 1 ; Iteration 6050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7692 ; Perplexity 26.71
[02/13/18 06:30:16 INFO] Epoch 1 ; Iteration 6100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7398 ; Perplexity 25.84
[02/13/18 06:31:33 INFO] Epoch 1 ; Iteration 6150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7356 ; Perplexity 25.89
[02/13/18 06:32:47 INFO] Epoch 1 ; Iteration 6200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7897 ; Perplexity 25.90
[02/13/18 06:34:02 INFO] Epoch 1 ; Iteration 6250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8202 ; Perplexity 26.13
[02/13/18 06:35:16 INFO] Epoch 1 ; Iteration 6300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8144 ; Perplexity 25.59
[02/13/18 06:36:31 INFO] Epoch 1 ; Iteration 6350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7607 ; Perplexity 24.70
[02/13/18 06:37:45 INFO] Epoch 1 ; Iteration 6400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7795 ; Perplexity 24.89
[02/13/18 06:39:00 INFO] Epoch 1 ; Iteration 6450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7856 ; Perplexity 24.74
[02/13/18 06:40:17 INFO] Epoch 1 ; Iteration 6500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7797 ; Perplexity 25.12
[02/13/18 06:41:33 INFO] Epoch 1 ; Iteration 6550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7837 ; Perplexity 24.56
[02/13/18 06:42:46 INFO] Epoch 1 ; Iteration 6600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7737 ; Perplexity 23.74
[02/13/18 06:44:03 INFO] Epoch 1 ; Iteration 6650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7762 ; Perplexity 24.32
[02/13/18 06:45:13 INFO] Epoch 1 ; Iteration 6700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8224 ; Perplexity 23.96
[02/13/18 06:46:29 INFO] Epoch 1 ; Iteration 6750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7844 ; Perplexity 24.53
[02/13/18 06:47:46 INFO] Epoch 1 ; Iteration 6800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7537 ; Perplexity 23.93
[02/13/18 06:49:03 INFO] Epoch 1 ; Iteration 6850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7901 ; Perplexity 23.86
[02/13/18 06:50:17 INFO] Epoch 1 ; Iteration 6900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7940 ; Perplexity 23.95
[02/13/18 06:51:34 INFO] Epoch 1 ; Iteration 6950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7929 ; Perplexity 23.80
[02/13/18 06:52:50 INFO] Epoch 1 ; Iteration 7000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7712 ; Perplexity 23.58
[02/13/18 06:54:02 INFO] Epoch 1 ; Iteration 7050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7933 ; Perplexity 22.70
[02/13/18 06:55:20 INFO] Epoch 1 ; Iteration 7100/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7479 ; Perplexity 23.21
[02/13/18 06:56:35 INFO] Epoch 1 ; Iteration 7150/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7717 ; Perplexity 22.46
[02/13/18 06:57:47 INFO] Epoch 1 ; Iteration 7200/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8006 ; Perplexity 22.71
[02/13/18 06:58:59 INFO] Epoch 1 ; Iteration 7250/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8104 ; Perplexity 22.77
[02/13/18 07:00:16 INFO] Epoch 1 ; Iteration 7300/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7603 ; Perplexity 22.22
[02/13/18 07:01:27 INFO] Epoch 1 ; Iteration 7350/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7900 ; Perplexity 22.45
[02/13/18 07:02:42 INFO] Epoch 1 ; Iteration 7400/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7785 ; Perplexity 22.28
[02/13/18 07:03:58 INFO] Epoch 1 ; Iteration 7450/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8118 ; Perplexity 22.62
[02/13/18 07:05:15 INFO] Epoch 1 ; Iteration 7500/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7528 ; Perplexity 22.35
[02/13/18 07:06:33 INFO] Epoch 1 ; Iteration 7550/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7624 ; Perplexity 22.47
[02/13/18 07:07:48 INFO] Epoch 1 ; Iteration 7600/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7961 ; Perplexity 22.16
[02/13/18 07:09:02 INFO] Epoch 1 ; Iteration 7650/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8066 ; Perplexity 21.83
[02/13/18 07:10:18 INFO] Epoch 1 ; Iteration 7700/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7746 ; Perplexity 21.72
[02/13/18 07:11:35 INFO] Epoch 1 ; Iteration 7750/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7816 ; Perplexity 21.71
[02/13/18 07:12:52 INFO] Epoch 1 ; Iteration 7800/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7574 ; Perplexity 20.93
[02/13/18 07:14:09 INFO] Epoch 1 ; Iteration 7850/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7757 ; Perplexity 21.45
[02/13/18 07:15:27 INFO] Epoch 1 ; Iteration 7900/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7555 ; Perplexity 21.41
[02/13/18 07:16:44 INFO] Epoch 1 ; Iteration 7950/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7666 ; Perplexity 21.49
[02/13/18 07:18:01 INFO] Epoch 1 ; Iteration 8000/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7547 ; Perplexity 21.35
[02/13/18 07:19:15 INFO] Epoch 1 ; Iteration 8050/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 8089 ; Perplexity 21.37
[02/13/18 07:20:29 INFO] Epoch 1 ; Iteration 8098/8098 ; Optim SGD LR 1.000000 ; Source tokens/s 7760 ; Perplexity 21.65
[02/13/18 07:20:29 INFO] Evaluating on the validation dataset...
[02/13/18 07:20:37 INFO] Validation perplexity: 17.75
[02/13/18 07:20:37 INFO] Saving checkpoint to '/home/benchmark01/wmt15-ende/wmt15-de-en/wmt15-all-en-de_epoch1_17.75.t7'...

real    256m49.284s
user    518m39.664s
sys     53m37.492s
Done: 15410 seconds elapsed.