Why does electra trained for 24h not perform well?

[jowagner]

Find out why we did not succeed in training a usable electra model with 24 hour computation budget in issue #76 when the 48 hour BERT models perform ok and electra is supposed to reach good performance levels much more quickly.

Reading:

• Chi et al. 2021 XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
• Anonymous (2021, ARR November submission) "BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla" train BERT using Electra