search

Alexander-H-Liu / End-to-end-ASR-Pytorch

This is an open source project (formerly named Listen, Attend and Spell - PyTorch Implementation) for end-to-end ASR implemented with Pytorch, the well known deep learning toolkit.
MIT License
1.19k stars 318 forks source link

Inference decoding procedure is biased #10

Closed b-etienne closed 5 years ago

b-etienne commented 5 years ago

I am not sure the decoding prodecure is right at test time. Indeed, you use max_label_len = min([batch_label.size()[1],kwargs['max_label_len']]) to tell the decoder when to stop. At test time, you don't feed the decoder with labels, so you shouldn't use labels lengths either, right ? In my case, I decode until all the items in the batch have emitted the EOS symbol, which means that the decoded sequences can be shorter or longer than the target sequences. As a consequence, you shouldn't be able to compute cross entropy loss on the test set...

Alexander-H-Liu commented 5 years ago

@b-etienne you're right, currently, the decoding procedure is only for validation, and that's why (to shorten run time of validation) However, I'll try to make validation less biased by adding constant additional steps to the decoding procedure in the next version. (and also a completely unbiased decoding function for test time) Thank you very much for pointing out this.

Alexander-H-Liu commented 5 years ago

Fixed during rebasing the project