WIP: 18.12 transformer fp16

NVIDIA / OpenSeq2Seq

Toolkit for efficient experimentation with Speech Recognition, Text2Speech and NLP

https://nvidia.github.io/OpenSeq2Seq

Apache License 2.0

1.54k stars 369 forks source link

Closed borisgin closed 5 years ago

borisgin commented 5 years ago

modified fp16 in transformer (removed up-conversion to float, attention mask in fp16)
added L1 support to transformer/layernorm
added batch_norm to transformer