asg criterion

[wbgxx333]

when i use the asg critertion instead of ctc,and the train is always training,but have no model and the log is null.could you help me ?

[jacobkahn]

@wbgxx333 You'll need to provide a lot more information if we're going to try to repro, including:

• Your platform/OS, criterion backend, and compiler
• The flags/flagsfile you're using.

In the mean time, can you try setting the reportiters flag to a small value so you can see if any iterations are occurring? ASG may in some cases be slightly slower to execute, which is why you might not be seeing output as frequently.

[MXuer]

@wbgxx333 You'll need to provide a lot more information if we're going to try to repro, including:

• Your platform/OS, criterion backend, and compiler
• The flags/flagsfile you're using.

In the mean time, can you try setting the reportiters flag to a small value so you can see if any iterations are occurring? ASG may in some cases be slightly slower to execute, which is why you might not be seeing output as frequently.

Hi, @jacobkahn our platform/OS is centOS, the backend is CUDA, and I tried setting reportiters=1, but the log is still null. And Here's my flagsfile:

Training config for Mini Librispeech

Replace [...] with appropriate paths

--datadir=/data/duhu/wav2letter/tutorials/aishell/s5 --tokensdir=/data/duhu/wav2letter/tutorials/aishell/s5 --rundir=/data/duhu/wav2letter/tutorials/aishell/s5 --archdir=/data/duhu/wav2letter/tutorials/aishell/ --train=data/train --valid=data/dev --input=wav --arch=network.arch --tokens=data/tokens.txt --criterion=asg --reportiters=1

--momentum=0.9

--weightdecay=0.0005

--lr=0.1 --stepsize=10 --gamma=0.9 --maxgradnorm=1.0 --replabel=2 --surround=| --onorm=target --sqnorm=true --mfsc=true --filterbanks=40 --nthread=4 --batchsize=4 --runname=aishell_lr_0.1_batch_4_decay_0.9_asg --iter=100

[jacobkahn]

So you be completely clear — you see logs when you train with CTC, but not with ASG in exactly the same environment? Do you see any log output at all when you train with ASG? If so, can you paste it here?

[MXuer]

@jacobkahn , Hi, both trainings with CTC and ASG were exactly the same environment. I just change the criterion from ctc to asg. It shows that the Trainer are always running, and the initialization is successful.However, nothing in the log. Here are another problem that the model's overfitting are very serious when I use CTC as the criterion. I add dropout=0.8 or the L2_penalty, both of them did not work. Is there any reason for this kind of phenomenon？ Thank you very much!

[jacobkahn]

@MXuer — can you check your GPU utilization during training to see if anything is happening?

[jacobkahn]

Issue resolved (#187).