search

tensorflow / models

Models and examples built with TensorFlow
Other
76.99k stars 45.78k forks source link

Documentation: ELECTRA Pretraining #11054

Open stefan-it opened 1 year ago

stefan-it commented 1 year ago

Dear TF Models Garden Maintainer,

today I recognized that TensorFlow 1.15 is no longer available as supported TPU version. That means, it is no longer possible to pretrain BERT and ELECTRA models with the official code bases.

As I was able to sucessfully pretrain Token Dropping BERT and TEAMS models (many thanks for open sourcing the implementation), I would like to know if it's already possible to pretrain "normal" ELECTRA models with TF Models Garden and if the necessary pretraining commands can be provided here and in the documentation.

Many thanks in advance!

stefan-it commented 1 year ago

Update: I could find BERT pretraining commands here:

https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrain.md

So only ELECTRA would be missing I think :hugs:

laxmareddyp commented 1 year ago

Hi @Chen-Zhe,

Could you please check this in your free time.

Thanks