The training has been running for 5 days, is this normal?

[hadilnc01]

This is the command I used to run the trainer: SentenceMatchTrainer.py --train_path train.tsv --dev_path dev.tsv --test_path test.tsv --fix_word_vec --model_dir models/ --MP_dim 10 --suffix sample --word_vec_path wordvec.txt

And these are the training results so far: Start the training loop. 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 Step 6405: loss = 3255.40 (81732.544 sec) Validation Data Eval: Current accuracy is 79.97 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 Step 12811: loss = 2481.54 (81754.701 sec) Validation Data Eval: Current accuracy is 83.78 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 Step 19217: loss = 2204.13 (28593.611 sec) Validation Data Eval: Current accuracy is 84.22 19300 19400 19500 19600 19700 19800 19900 20000 20100 20200 20300 20400 20500 20600 20700 20800 20900 21000 21100 21200 21300 21400 21500 21600 21700 21800 21900 22000 22100 22200 22300 22400 22500 22600 22700 22800 22900 23000 23100 23200 23300 23400 23500 23600 23700 23800 23900 24000 24100 24200 24300 24400 24500 24600 24700 24800 24900 25000 25100 25200 25300 25400 25500 25600 Step 25623: loss = 2029.37 (296247.130 sec) Validation Data Eval: Current accuracy is 85.67 25700 25800 25900 26000 26100 26200 26300 26400 26500 26600 26700 26800 26900 27000 27100 27200 27300

Will it make 10 rounds of this because I set the --MP_dim parameter to 10? Or am I missing something?

[zhiguowang]

Here is one of the the log files in my experiments. quora_exp.txt

You experiment looks very slow. Did you use any GPU device?

[hadilnc01]

I just ran it on my mac, I didn't use any GPU device. I'm fine with it taking time but I just wanted to make sure if it has to do 10 rounds to be done. I wanted to make sure that it won't keep running until I explicitly stop it. Based on your log file, I can see that you set the MP_dim to be 10 as well and you had a total of 10 training rounds so I think the same case applies to me. I did 6 rounds so far so 4 more to go.