Add a BertMaskedLM task model

keras-team / keras-nlp

Modular Natural Language Processing workflows with Keras

Apache License 2.0

763 stars 228 forks source link

Closed mattdangerw closed 1 year ago

mattdangerw commented 1 year ago

In https://github.com/keras-team/keras-nlp/pull/653 we added a masked language modeling task for RoBERTa. We can make a similar change for the BERT model.

[ ] Update BertTokenizer to expect a mask token.
[ ] Add a BertMaskedLMPreprocessor preprocessor layer and tests.
[ ] Add a BertMaskedLM task model and tests.
[ ] Update keras_nlp/models/__init__.py to export BertMaskedLM and BertMaskedLMPreprocessor.

Cyber-Machine commented 1 year ago

@mattdangerw I would like to work on this issue!

shivance commented 1 year ago

@Cyber-Machine are you still working on it ? If not, can I take it?

Cyber-Machine commented 1 year ago

@shivance Yes, I am still working in this issue.

jayam30 commented 1 year ago

Hey! I would like to take this issue up, if others are not @Cyber-Machine

Cyber-Machine commented 1 year ago

@jayam30 I am currently working on it. Will be raising PR by tonight, it's ready and I have to add tests.