search

srvk / eesen

The official repository of the Eesen project
http://arxiv.org/abs/1507.08240
Apache License 2.0
822 stars 342 forks source link

ctc loss output is weird with training in cpu only #31

Closed jinserk closed 8 years ago

jinserk commented 8 years ago

Hi, Since the GPU nodes are being in maintenance, I need to compile the code with --use-cuda=no option. However, there were a few compile errors, especially related to ExpA, so I changed them to exp. By the way, when the EESEN enters into the training phase, the log show like this: train-ctc-parallel --report-step=1000 --num-sequence=10 --frame-limit=25000 --learn-rate=0.00004 --momentum=0.9 --verbose=1 'ark,s,cs:copy-feats scp:exp/train_phn_l4_c320/train_local.scp ark:- | add-deltas ark:- ark:- |' 'ark:gunzip -c exp/train_phn_l4_c320/labels.tr.gz|' exp/train_phn_l4_c320/nnet/nnet.iter0 exp/train_phn_l4_c320/nnet/nnet.iter1 copy-feats scp:exp/train_phn_l4_c320/train_local.scp ark:- add-deltas ark:- ark:- LOG (train-ctc-parallel:main():train-ctc-parallel.cc:112) TRAINING STARTED VLOG1 After 1010 sequences (0.695456Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -36.2847% VLOG1 After 2020 sequences (1.62006Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -30.1322% VLOG1 After 3030 sequences (2.67991Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -25.7895% VLOG1 After 4040 sequences (3.84653Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -23.9529% VLOG1 After 5050 sequences (5.10758Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -23.4412% VLOG1 After 6060 sequences (6.45321Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -22.1382% VLOG1 After 7070 sequences (7.87719Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -20.9931% VLOG1 After 8080 sequences (9.37634Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -20.5369% VLOG1 After 9090 sequences (10.944Hr): Obj(log[Pzx]) = -1e+30 TokenAcc = -20.2233%

Is this normal? What does the negative accuracy mean? Thank you.

Jinserk

yajiemiao commented 8 years ago

We just checked in fixes to Eesen that would solve the CPU compilation issue. Please update to the latest. Please stick to the original ExpA() function (rather than exp()) which deals with some critical extreme cases. CPU-based CTC training is NOT available in Eesen. You have to rely on GPUs for training.

jinserk commented 8 years ago

Thank you for reply. I confirmed the updated code is compiled well in CPU only mode. By the way, is there any reason the CTC cost function doesn't support the CPU only mode? Too slow?

yajiemiao commented 8 years ago

Yes, for most scenarios, CPU training is not useful, given the data size of ASR tasks.