A question about `hidden_act`

google-research / albert

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Apache License 2.0

3.24k stars 569 forks source link

A question about `hidden_act` #154

Closed beamind closed 4 years ago

beamind commented 4 years ago

According to part 3.1 of paper, the hidde_act of albert model is GELU. But according to albert_config.json file from albert_xxlarge_zh.tar.gz, the hidden_act is RELU. which is correct? Thanks!

Danny-Google commented 4 years ago

We use RLEU for Chinese version and GELU for English eversion