Transformer model recipe for Librispeech is not working

[PhenixCFLi]

🐛 Bug

Failed running the librispeech training with speech_transformer_librispeech architecture.

To Reproduce Steps to reproduce the behavior (always include the command you ran):

When I change the script "run.sh" under folder "/examples/asr_librispeech" to use arch "speech_transformer_librispeech",.

I changed the "run.sh" on below lines:

# Just start training and skip other preparation process
# stage=1 
stage=8

and

# Change arch from speech_conv_lstm_librispeech to speech_transformer_librispeech
  CUDA_VISIBLE_DEVICES=$free_gpu speech_train.py data --task speech_recognition_espresso --seed 1 --user-dir espresso \
    --num-workers 0 --data-buffer-size 0 --max-tokens 26000 --max-sentences 24 --curriculum 1 \
    --valid-subset $valid_subset --max-sentences-valid 48 --ddp-backend no_c10d \
    --distributed-world-size $ngpus --distributed-port $(if [ $ngpus -gt 1 ]; then echo 100; else echo -1; fi) \
    --optimizer adam --lr 0.001 --weight-decay 0.0 --clip-norm 2.0 \
    --save-dir $dir --restore-file checkpoint_last.pt --save-interval-updates $((6000/ngpus)) \
    --keep-interval-updates 3 --keep-last-epochs 5 --validate-interval 1 --best-checkpoint-metric wer \
    --dict $dict --bpe sentencepiece --sentencepiece-vocab ${sentencepiece_model}.model \
    --max-source-positions 9999 --max-target-positions 999 \
    --log-interval $((8000/ngpus)) --log-format simple \
    --arch **speech_transformer_librispeech** --criterion cross_entropy_v2 \
    --print-training-sample-interval $((4000/ngpus)) \
    $opts --specaugment-config "$specaug_config" 2>&1 | tee $log_file

Run cmd './run.sh' Got error

Traceback (most recent call last):
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/speech_train.py", line 341, in distributed_main
    main(args, init_distributed=True)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/speech_train.py", line 72, in main
    model = task.build_model(args)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/tasks/speech_recognition.py", line 339, in build_model
    model = super().build_model(args)
  File "/nfs/mercury-13/u20/cli/src/espresso/fairseq/tasks/fairseq_task.py", line 211, in build_model
    model = models.build_model(args, self)
  File "/nfs/mercury-13/u20/cli/src/espresso/fairseq/models/__init__.py", line 48, in build_model
    return ARCH_MODEL_REGISTRY[args.arch].build_model(args, task)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/models/speech_transformer.py", line 132, in build_model
    return cls(encoder, decoder)
TypeError: __init__() missing 1 required positional argument: 'decoder'

This is due to the constructor not insert with args, and can be fixed by add back the args param in speech_transformer.py as below

        # return cls(encoder, decoder)
        return cls(args, encoder, decoder)

But after that, it got more complicated error following as below

-- Process 1 terminated with the following error:
Traceback (most recent call last):
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/speech_train.py", line 341, in distributed_main
    main(args, init_distributed=True)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/speech_train.py", line 121, in main
    valid_losses, should_stop = train(args, trainer, task, epoch_itr)
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/speech_train.py", line 210, in train
    log_output = trainer.train_step(samples)
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/contextlib.py", line 74, in inner
    return func(*args, **kwds)
  File "/nfs/mercury-13/u20/cli/src/espresso/fairseq/trainer.py", line 408, in train_step
    ignore_grad=is_dummy_batch,
  File "/nfs/mercury-13/u20/cli/src/espresso/fairseq/tasks/fairseq_task.py", line 342, in train_step
    loss, sample_size, logging_output = criterion(model, sample)
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/nfs/mercury-13/u20/cli/src/espresso/espresso/criterions/cross_entropy_v2.py", line 49, in forward
    net_output = model(**sample["net_input"], epoch=self.epoch)
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/nfs/mercury-13/u20/cli/src/espresso/fairseq/legacy_distributed_data_parallel.py", line 86, in forward
    return self.module(*inputs, **kwargs)
  File "/nfs/mercury-13/u20/cli/miniconda3/envs/espresso/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'epoch'

[freewym]

Are you using the latest version? The latest one shouldn't have such problem.

BTW, actually I haven't made the transformer model on LibriSpeech converge, and I am still tuning the recipe. But swbd and WSJ's Transformer recipes should work very well.

[PhenixCFLi]

Thanks a lot for your help. I have updated to latest version and training started successfully now ... 👍