What's the hyperparameters to finetune Chinese bert-large on CLUE?

dbiir / UER-py

Open Source Pre-training Model Framework in PyTorch & Pre-trained Model Zoo

https://github.com/dbiir/UER-py/wiki

Apache License 2.0

2.96k stars 524 forks source link

What's the hyperparameters to finetune Chinese bert-large on CLUE? #44

Open hitvoice opened 4 years ago

hitvoice commented 4 years ago

Recently you updated "BERT pretrained on mixed large Chinese corpus (bert-large 24-layers) " on ReadMe. What hyperparameters (lr, batch size, max epochs) did you use when fine-tuning on CLUE?

zhezhaoa commented 4 years ago

We basically use the following setting: lr=2e-5 batch_size=32, max_epochs=3 To reproduce the results on CLUE, we also need some other techniques which has been listed on CLUE leaderboard. We will open-source related code in the near future.

hitvoice commented 4 years ago

Got it. Thanks for answering!