Implement FastEmit regularization

1ytic / warp-rnnt

CUDA-Warp RNN-Transducer

MIT License

211 stars 41 forks source link

Implement FastEmit regularization #23

Closed iceychris closed 3 years ago

iceychris commented 3 years ago

Hey!

Thank you for this great library!

This PR implements FastEmit regularization from https://arxiv.org/abs/2010.11148. The gradients of all non-blank symbols are scaled up by a small factor (around 1.004). Intuitively, the model is encouraged to output symbols faster, thus reducing latency.

1ytic commented 3 years ago

Thank you! Looks simple. Unfortunately I will not be able to test the final streaming performance, but I will happily add this code. Could you fix core_gather.cu as well? This is the most efficient implementation and should be used by default.

iceychris commented 3 years ago

Nice, thank you!