search

jbrry / Irish-BERT

Repository to store helper scripts for creating an Irish BERT model.
Other
9 stars 0 forks source link

Paper: report random seeds of from scratch models #104

Closed jowagner closed 2 years ago

jowagner commented 2 years ago

Say whether each model tried during development used a new seed for initialisation or the same seed throughout. List seed(s) in appendix.

There appears to be no call of tensorflow's set_seed() function in BERT's run_pretraining.py or any of the other .py files in the bert repo. The bert_config.json also does not configure a seed. There is a default seed 12345 for the creation of the binary pretraining data. However, this should not affect the initialisation of the model. (It will affect training outcomes. If we want to try different seeds in future work we should change both the data seed and the model seed.)

Wikibert's sampledocs.py has a --seed option but it is None by default, setting the initial state of the PRNG from a non-deterministic system source of randomness.

The only mention of seeds in the BERT paper, Devlin et al. (2019), is for fine-tuning.

jowagner commented 2 years ago

Described in middle of section 3.3