search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
134.73k stars 26.94k forks source link

can this project select the specific version of BERT? #1076

Closed bytekongfrombupt closed 5 years ago

bytekongfrombupt commented 5 years ago

❓ Questions & Help

I dont know if this project can select the version of BERT which I need. For example, i want use BERT-wwm not BERT-basic, what should i do？ Can you help me, plz.

LysandreJik commented 5 years ago

Hi. You can check the documentation about the different checkpoints available for each model here.

If you're looking for BERT whole word masking, there are the following pretrained models that might be of interest: bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad and bert-large-cased-whole-word-masking-finetuned-squad.

bytekongfrombupt commented 5 years ago

@LysandreJik Thanks for your advice. But, in my situation, I have to use my corpus to train a new BERT with whole word mask, so I cant use the pre-trained BERT model, what I should do in this situation?

LysandreJik commented 5 years ago

Training an entire BERT model from scratch takes a lot of resources, and we don't have any scripts/examples that show how to do it with our library.

You could look at Microsoft's repository that uses our implementation to pre-train/fine-tune BERT.

bytekongfrombupt commented 5 years ago

@LysandreJik notice that the bert had been updated by adding whole word mask, do you updating your pytorch-transformer with this trick when you convert BERT from tf to pytorch