search

mravanelli / pytorch-kaldi

pytorch-kaldi is a project for developing state-of-the-art DNN/RNN hybrid speech recognition systems. The DNN part is managed by pytorch, while feature extraction, label computation, and decoding are performed with the kaldi toolkit.
2.37k stars 446 forks source link

The loss curve of train and dev is reasonable but why the Test Error keeps 53% or so? #244

Closed hellboywyh closed 4 years ago

hellboywyh commented 4 years ago

image image It has confused me for many cfg files. I'd appreciate it if someone can offer some help.

TParcollet commented 4 years ago

Hi there, we will need a bit more of context here. What is the experiment that you are running ?

hellboywyh commented 4 years ago

Command

python run_exp.py cfg/TIMIT_baselines/TIMIT_MLP_mfcc_basic.cfg

Here are the full output of this command.

root@9fl0tel6v6f7a-0:/yhwang/0-Projects/0-kaldi-lstm/1-pytorch-kaldi# python run_exp.py cfg/TIMIT_baselines/TIMIT_MLP_mfcc_basic.cfg 
- Reading config file......OK!
- Chunk creation......OK!

------------------------------ Epoch 00 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1776/1776)) | L:4.259
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1752/1752)) | L:2.881
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1727/1727)) | L:2.482
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1740/1740)) | L:2.329
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1790/1790)) | L:2.217
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 00 / 23
Training on ['TIMIT_tr']
Loss = 2.833 | err = 0.668 
-----
Validating on TIMIT_dev
Loss = 2.163 | err = 0.575 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 55

------------------------------ Epoch 01 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1752/1752)) | L:2.042
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1775/1775)) | L:2.005
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1732/1732)) | L:1.984
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1751/1751)) | L:1.941
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:1.951
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 01 / 23
Training on ['TIMIT_tr']
Loss = 1.984 | err = 0.543 
-----
Validating on TIMIT_dev
Loss = 1.953 | err = 0.532 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 53

------------------------------ Epoch 02 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1771/1771)) | L:1.764
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1750/1750)) | L:1.801
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1736/1736)) | L:1.779
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1746/1746)) | L:1.784
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1781/1781)) | L:1.782
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 02 / 23
Training on ['TIMIT_tr']
Loss = 1.782 | err = 0.502 
-----
Validating on TIMIT_dev
Loss = 1.852 | err = 0.510 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 54

------------------------------ Epoch 03 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1761/1761)) | L:1.609
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1743/1743)) | L:1.648
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1760/1760)) | L:1.674
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1746/1746)) | L:1.667
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:1.671
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 03 / 23
Training on ['TIMIT_tr']
Loss = 1.653 | err = 0.474 
-----
Validating on TIMIT_dev
Loss = 1.807 | err = 0.499 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 54

------------------------------ Epoch 04 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1767/1767)) | L:1.531
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1748/1748)) | L:1.563
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1745/1745)) | L:1.566
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1747/1747)) | L:1.567
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1777/1777)) | L:1.571
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 04 / 23
Training on ['TIMIT_tr']
Loss = 1.559 | err = 0.453 
-----
Validating on TIMIT_dev
Loss = 1.776 | err = 0.492 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 53

------------------------------ Epoch 05 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1739/1739)) | L:1.444
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1770/1770)) | L:1.459
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1793/1793)) | L:1.509
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1742/1742)) | L:1.507
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1742/1742)) | L:1.492
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 05 / 23
Training on ['TIMIT_tr']
Loss = 1.482 | err = 0.435 
-----
Validating on TIMIT_dev
Loss = 1.740 | err = 0.482 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 51

------------------------------ Epoch 06 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1744/1744)) | L:1.364
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:1.415
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1771/1771)) | L:1.418
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1744/1744)) | L:1.439
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1753/1753)) | L:1.442
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 06 / 23
Training on ['TIMIT_tr']
Loss = 1.416 | err = 0.419 
-----
Validating on TIMIT_dev
Loss = 1.738 | err = 0.479 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 53

------------------------------ Epoch 07 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1722/1722)) | L:1.293
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1757/1757)) | L:1.354
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1766/1766)) | L:1.378
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1785/1785)) | L:1.377
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1756/1756)) | L:1.394
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 07 / 23
Training on ['TIMIT_tr']
Loss = 1.359 | err = 0.406 
-----
Validating on TIMIT_dev
Loss = 1.725 | err = 0.474 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 54

------------------------------ Epoch 08 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1736/1736)) | L:1.264
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1781/1781)) | L:1.307
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1772/1772)) | L:1.295
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1742/1742)) | L:1.336
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1754/1754)) | L:1.348
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 08 / 23
Training on ['TIMIT_tr']
Loss = 1.310 | err = 0.395 
-----
Validating on TIMIT_dev
Loss = 1.722 | err = 0.471 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 53

------------------------------ Epoch 09 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1768/1768)) | L:1.213
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1772/1772)) | L:1.255
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1751/1751)) | L:1.279
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1740/1740)) | L:1.291
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1753/1753)) | L:1.305
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 09 / 23
Training on ['TIMIT_tr']
Loss = 1.269 | err = 0.384 
-----
Validating on TIMIT_dev
Loss = 1.722 | err = 0.470 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 52

------------------------------ Epoch 10 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1743/1743)) | L:1.161
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1758/1758)) | L:1.202
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1761/1761)) | L:1.216
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1806/1806)) | L:1.273
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1717/1717)) | L:1.276
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 10 / 23
Training on ['TIMIT_tr']
Loss = 1.226 | err = 0.374 
-----
Validating on TIMIT_dev
Loss = 1.728 | err = 0.472 
-----
Learning rate on architecture1 = 0.08 
-----
Elapsed time (s) = 52

------------------------------ Epoch 11 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1781/1781)) | L:1.109
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1738/1738)) | L:1.126
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1747/1747)) | L:1.122
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:1.141
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1745/1745)) | L:1.151
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 11 / 23
Training on ['TIMIT_tr']
Loss = 1.130 | err = 0.348 
-----
Validating on TIMIT_dev
Loss = 1.667 | err = 0.457 
-----
Learning rate on architecture1 = 0.04 
-----
Elapsed time (s) = 54

------------------------------ Epoch 12 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1760/1760)) | L:1.043
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1780/1780)) | L:1.093
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1717/1717)) | L:1.085
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1783/1783)) | L:1.102
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1744/1744)) | L:1.121
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 12 / 23
Training on ['TIMIT_tr']
Loss = 1.089 | err = 0.338 
-----
Validating on TIMIT_dev
Loss = 1.676 | err = 0.455 
-----
Learning rate on architecture1 = 0.04 
-----
Elapsed time (s) = 54

------------------------------ Epoch 13 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1767/1767)) | L:1.014
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1728/1728)) | L:1.048
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1759/1759)) | L:1.084
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1779/1779)) | L:1.081
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1752/1752)) | L:1.097
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 13 / 23
Training on ['TIMIT_tr']
Loss = 1.065 | err = 0.332 
-----
Validating on TIMIT_dev
Loss = 1.681 | err = 0.456 
-----
Learning rate on architecture1 = 0.04 
-----
Elapsed time (s) = 54

------------------------------ Epoch 14 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1768/1768)) | L:1.007
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1738/1738)) | L:1.019
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1768/1768)) | L:1.003
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1749/1749)) | L:1.018
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1763/1763)) | L:1.031
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 14 / 23
Training on ['TIMIT_tr']
Loss = 1.015 | err = 0.318 
-----
Validating on TIMIT_dev
Loss = 1.658 | err = 0.447 
-----
Learning rate on architecture1 = 0.02 
-----
Elapsed time (s) = 50

------------------------------ Epoch 15 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1744/1744)) | L:0.961
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1754/1754)) | L:0.965
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1758/1758)) | L:1.008
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1773/1773)) | L:1.005
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1756/1756)) | L:1.024
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 15 / 23
Training on ['TIMIT_tr']
Loss = 0.993 | err = 0.311 
-----
Validating on TIMIT_dev
Loss = 1.669 | err = 0.450 
-----
Learning rate on architecture1 = 0.02 
-----
Elapsed time (s) = 53

------------------------------ Epoch 16 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1738/1738)) | L:0.959
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1766/1766)) | L:0.959
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1757/1757)) | L:0.964
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1748/1748)) | L:0.968
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1775/1775)) | L:0.984
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 16 / 23
Training on ['TIMIT_tr']
Loss = 0.967 | err = 0.304 
-----
Validating on TIMIT_dev
Loss = 1.656 | err = 0.447 
-----
Learning rate on architecture1 = 0.01 
-----
Elapsed time (s) = 52

------------------------------ Epoch 17 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1773/1773)) | L:0.935
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1728/1728)) | L:0.947
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:0.969
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1746/1746)) | L:0.955
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1764/1764)) | L:0.977
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 17 / 23
Training on ['TIMIT_tr']
Loss = 0.955 | err = 0.300 
-----
Validating on TIMIT_dev
Loss = 1.655 | err = 0.446 
-----
Learning rate on architecture1 = 0.01 
-----
Elapsed time (s) = 51

------------------------------ Epoch 18 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1747/1747)) | L:0.931
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1741/1741)) | L:0.941
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1716/1716)) | L:0.938
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1793/1793)) | L:0.955
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1788/1788)) | L:0.963
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 18 / 23
Training on ['TIMIT_tr']
Loss = 0.945 | err = 0.297 
-----
Validating on TIMIT_dev
Loss = 1.660 | err = 0.447 
-----
Learning rate on architecture1 = 0.01 
-----
Elapsed time (s) = 54

------------------------------ Epoch 19 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1760/1760)) | L:0.924
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1767/1767)) | L:0.942
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1766/1766)) | L:0.919
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1753/1753)) | L:0.934
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1739/1739)) | L:0.943
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 19 / 23
Training on ['TIMIT_tr']
Loss = 0.932 | err = 0.294 
-----
Validating on TIMIT_dev
Loss = 1.655 | err = 0.445 
-----
Learning rate on architecture1 = 0.005 
-----
Elapsed time (s) = 51

------------------------------ Epoch 20 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1758/1758)) | L:0.907
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1752/1752)) | L:0.924
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1750/1750)) | L:0.928
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1761/1761)) | L:0.922
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1764/1764)) | L:0.943
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 20 / 23
Training on ['TIMIT_tr']
Loss = 0.925 | err = 0.292 
-----
Validating on TIMIT_dev
Loss = 1.659 | err = 0.445 
-----
Learning rate on architecture1 = 0.005 
-----
Elapsed time (s) = 54

------------------------------ Epoch 21 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1744/1744)) | L:0.919
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1754/1754)) | L:0.917
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1778/1778)) | L:0.907
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1766/1766)) | L:0.939
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1743/1743)) | L:0.931
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 21 / 23
Training on ['TIMIT_tr']
Loss = 0.919 | err = 0.290 
-----
Validating on TIMIT_dev
Loss = 1.654 | err = 0.444 
-----
Learning rate on architecture1 = 0.0025 
-----
Elapsed time (s) = 55

------------------------------ Epoch 22 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1717/1717)) | L:0.911
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:0.911
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1787/1787)) | L:0.915
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:0.935
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1733/1733)) | L:0.911
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 22 / 23
Training on ['TIMIT_tr']
Loss = 0.916 | err = 0.289 
-----
Validating on TIMIT_dev
Loss = 1.653 | err = 0.445 
-----
Learning rate on architecture1 = 0.0025 
-----
Elapsed time (s) = 53

------------------------------ Epoch 23 / 23 ------------------------------
Training TIMIT_tr chunk = 1 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1759/1759)) | L:0.923
Training TIMIT_tr chunk = 2 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1759/1759)) | L:0.904
Training TIMIT_tr chunk = 3 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1745/1745)) | L:0.911
Training TIMIT_tr chunk = 4 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1774/1774)) | L:0.912
Training TIMIT_tr chunk = 5 / 5
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Training | (Batch 1748/1748)) | L:0.916
Validating TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/test/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Validating | (Batch 956/956))

----- Summary epoch 23 / 23
Training on ['TIMIT_tr']
Loss = 0.913 | err = 0.288 
-----
Validating on TIMIT_dev
Loss = 1.652 | err = 0.444 
-----
Learning rate on architecture1 = 0.00125 
-----
Elapsed time (s) = 54

Testing TIMIT_test chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/test/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Forwarding | (Batch 192/192))
Decoding TIMIT_test output out_dnn1
%WER 51.5 | 192 7215 | 52.9 27.2 19.9 4.4 51.5 100.0 | -1.482 | /yhwang/0-Projects/0-kaldi-lstm/1-pytorch-kaldi/exp/TIMIT_MLP_mfcc_basic/decode_TIMIT_test_out_dnn1/score_7/ctm_39phn.filt.sys

-----
Generating output files and plots ... 
OK
timolohrenz commented 4 years ago

Can you maybe try to also do decoding on the TIMIT_dev data? Simply add it to the forward_with option in your config (forward_with =TIMIT_dev,TIMIT_test)

This helps narrowing down if this issue is related to decoding or not. Sharing the .cfg file here would also help.

Best regards, Timo

hellboywyh commented 4 years ago

Here is my cfg file:

[cfg_proto]
cfg_proto = proto/global.proto
cfg_proto_chunk = proto/global_chunk.proto

[exp]
cmd = 
run_nn_script = run_nn
out_folder = exp/TIMIT_MLP_basic
seed = 1234
use_cuda = True
multi_gpu = False
save_gpumem = False
n_epochs_tr = 24

[dataset1]
data_name = TIMIT_tr
fea = fea_name=mfcc
    fea_lst=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/train/feats.scp
    fea_opts=apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/train/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_train.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
    cw_left=5
    cw_right=5

lab = lab_name=lab_cd
    lab_folder=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali
    lab_opts=ali-to-pdf
    lab_count_file=auto
    lab_data_folder=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/train/
    lab_graph=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/exp/tri3/graph

n_chunks = 5

[dataset2]
data_name = TIMIT_dev
fea = fea_name=mfcc
    fea_lst=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/dev/feats.scp
    fea_opts=apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
    cw_left=5
    cw_right=5

lab = lab_name=lab_cd
    lab_folder=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali_dev
    lab_opts=ali-to-pdf
    lab_count_file=auto
    lab_data_folder=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/dev/
    lab_graph=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/exp/tri3/graph

n_chunks = 1

[dataset3]
data_name = TIMIT_test
fea = fea_name=mfcc
    fea_lst=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/test/feats.scp
    fea_opts=apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/test/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
    cw_left=5
    cw_right=5

lab = lab_name=lab_cd
    lab_folder=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/exp/dnn4_pretrain-dbn_dnn_ali_test
    lab_opts=ali-to-pdf
    lab_count_file=auto
    lab_data_folder=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data/test/
    lab_graph=/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/exp/tri3/graph

n_chunks = 1

[data_use]
train_with = TIMIT_tr
valid_with = TIMIT_dev
forward_with = TIMIT_dev,TIMIT_test

[batches]
batch_size_train = 128
max_seq_length_train = 1000
increase_seq_length_train = False
start_seq_len_train = 100
multply_factor_seq_len_train = 2
batch_size_valid = 128
max_seq_length_valid = 1000

[architecture1]
arch_name = MLP_layers1
arch_proto = proto/MLP.proto
arch_library = neural_networks
arch_class = MLP
arch_pretrain_file = none
arch_freeze = False
arch_seq_model = False
dnn_lay = 1024,1024,1024,1024,N_out_lab_cd
dnn_drop = 0.15,0.15,0.15,0.15,0.0
dnn_use_laynorm_inp = False
dnn_use_batchnorm_inp = False
dnn_use_batchnorm = True,True,True,True,False
dnn_use_laynorm = False,False,False,False,False
dnn_act = relu,relu,relu,relu,softmax
arch_lr = 0.08
arch_halving_factor = 0.5
arch_improvement_threshold = 0.001
arch_opt = sgd
opt_momentum = 0.0
opt_weight_decay = 0.0
opt_dampening = 0.0
opt_nesterov = False

[model]
model_proto = proto/model.proto
model = out_dnn1=compute(MLP_layers1,mfcc)
    loss_final=cost_nll(out_dnn1,lab_cd)
    err_final=cost_err(out_dnn1,lab_cd)

[forward]
forward_out = out_dnn1
normalize_posteriors = True
normalize_with_counts_from = lab_cd
save_out_file = False
require_decoding = True

[decoding]
decoding_script_folder = kaldi_decoding_scripts/
decoding_script = decode_dnn.sh
decoding_proto = proto/decoding.proto
min_active = 200
max_active = 7000
max_mem = 50000000
beam = 13.0
latbeam = 8.0
acwt = 0.2
max_arcs = -1
skip_scoring = false
scoring_script = local/score.sh
scoring_opts = "--min-lmwt 1 --max-lmwt 10"
norm_vars = False
timolohrenz commented 4 years ago

Thanks for that info. Even though 55% on TIMIT test set is quite high, I am not sure what PER we can expect for such a basic network on TIMIT. Please tell us what PER the dev-set decoding yields when it is done.

hellboywyh commented 4 years ago

It should be 18.3% PER according to the README.md file(Tutorials-TIMIT tutorial-7.Run the ASR experiment), and I have realized it one time several days before...

Here are the PER of the dev-set:


Testing TIMIT_dev chunk = 1 / 1
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/dev/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_dev.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
apply-cmvn --utt2spk=ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/data-mfcc/test/utt2spk  ark:/yhwang/0-Projects/0-kaldi-lstm/0-kaldi/egs/timit/s5/mfcc/cmvn_test.ark ark:- ark:- | add-deltas --delta-order=2 ark:- ark:- |
[========================================] 100% Forwarding | (Batch 400/400))
Testing TIMIT_test chunk = 1 / 1
Decoding TIMIT_dev output out_dnn1
%WER 51.5 | 400 15057 | 52.4 25.8 21.8 4.0 51.5 100.0 | -1.587 | /yhwang/0-Projects/0-kaldi-lstm/1-pytorch-kaldi/exp/TIMIT_MLP_basic/decode_TIMIT_dev_out_dnn1/score_10/ctm_39phn.filt.sys

Decoding TIMIT_test output out_dnn1
%WER 51.5 | 192 7215 | 52.9 27.2 19.9 4.4 51.5 100.0 | -1.482 | /yhwang/0-Projects/0-kaldi-lstm/1-pytorch-kaldi/exp/TIMIT_MLP_basic/decode_TIMIT_test_out_dnn1/score_7/ctm_39phn.filt.sys

I am more confused now....The dev WER is high, too... And I have another question. I notice that during training, it caculates the err of train-set and dev-set. Why not use WER? And how can I get the err of test-dev? Thank you!!

timolohrenz commented 4 years ago

Hey, sorry for the late answer. So that's a hint that something wents wrong during decoding. Reporting the WER for each epoch would indeed be handy, but it takes a lot computational effort as it includes decoding with language models etc. While for TIMIT this might be feasible it would slow down training a lot. You can look at End2End toolkits and approaches that are able to do that at the cost of flexibility.

Back to your issue I guess it might make sense to look into the forward out data from pytorch-kaldi which is written into the exp directory. When you use the kaldi cmd copy-feats you can transform them to human readable matrices and check their dimensionality and if they are proper likelihoods.

Next thing is to check the scoring. Look into the phoneme hypotheses outputs from kaldi decoding and if they correspond to the ground truth they are compared to. On TIMIT its sometimes an issue if different-sized phoneme sets are used.

Did you make any modifications from Kaldi's standard TIMIT recipe when you generated labels and features?

Hope I could help...

Best regards, Timo

hellboywyh commented 4 years ago

The problem has been solved. Thanks a lot! It may caused by the mfcc, fbank and fmllr feature generation process. I generate them in sequence before. But now adding a more mfcc feature generation process in the end, it works well.