search

kpe / bert-for-tf2

A Keras TensorFlow 2.0 implementation of BERT, ALBERT and adapter-BERT.
https://github.com/kpe/bert-for-tf2
MIT License
802 stars 193 forks source link

mixed precision #77

Open caishiqing opened 4 years ago

caishiqing commented 4 years ago

It seems that the package dose not support mixed precision training, since the Embedding layer needs add operating with output and embedding parameters, which dtype are 'float32' and 'float16' leading to a dtype error "TypeError: Input 'y' of 'AddV2' Op has type float16 that does not match type float32 of argument 'x'.". Is there any plane to fix this?

jeisinge commented 4 years ago

It is interesting to note that we were able to use mixed-precision training before with TF 1.15 and TF Estimator. With TF 2.3 and Keras, we are seeing this same error.

callzhang commented 3 years ago

Any updates on this?

leviswind commented 3 years ago

Any updates on this?