Pretraining iterations_per_loop and batch_size

google-research / albert

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Apache License 2.0

3.23k stars 570 forks source link

Pretraining iterations_per_loop and batch_size #179

Open likith187 opened 4 years ago

likith187 commented 4 years ago

I'm running this on colab. What iterations_per_loop should I set it to and the corresponding batch_size. Colab has no problems taking 5000 iterations_per_loop. what are the most optimal parameters to run on one colab tpu?