search

keras-team / keras-nlp

Modular Natural Language Processing workflows with Keras
Apache License 2.0
740 stars 218 forks source link

Add a keras.io guide for pretraining a transformer with keras-nlp #130

Open mattdangerw opened 2 years ago

mattdangerw commented 2 years ago

Working on a keras.io guide for pretraining a keras-nlp transformer model from scratch, using word piece tokenizer, transformer encoder, embedding layers, and our MLM layer helpers.

Will link a draft of this for keras.io when it is up, opening this so we can track for the 0.2.0 release.

mattdangerw commented 2 years ago

Guide is incoming https://github.com/keras-team/keras-io/pull/859

ddofer commented 1 year ago

Is it maybe relevant to add "load a pretrained (BERT) model" to this task?

mattdangerw commented 1 year ago

@ddofer this is incoming! And top priority for us actually.