search

OpenMined / SyferText

A privacy preserving NLP framework
Apache License 2.0
197 stars 49 forks source link

BERT MLM Support #208

Open ArjunParthasarathy opened 3 years ago

ArjunParthasarathy commented 3 years ago

Description

@AlanAboudib this adds support for a BERT encoder and iterator, specifically for the Masked LM use case.

Affected Dependencies

Now requires HuggingFace Transformers library to be installed.

How has this been tested?

I used a structure very similar to the BPTT Example Notebook to verify that my encoder and iterator work in training a BERT model. I was able to get good test and validation scores for my trained model on the Wikitext-2 dataset.

review-notebook-app[bot] commented 3 years ago

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB