search

microsoft / AzureML-BERT

End-to-End recipes for pre-training and fine-tuning BERT using Azure Machine Learning Service
https://azure.microsoft.com/en-us/blog/microsoft-makes-it-easier-to-build-popular-language-representation-model-bert-at-large-scale/
MIT License
391 stars 127 forks source link

Expected pretraining results (e.g., loss, next-sentence-prediction accuracy, etc.) #46

Open usuyama opened 4 years ago

usuyama commented 4 years ago

Would be great to get expected pretraining results (e.g., loss, next-sentence-prediction accuracy, etc.) and learning curves using the 64 * V100 training.

I found the fine-tuning task results on the blog but would be nice to compare the pretraining results as a "sanity check."

I don't have access to V100 GPUs for now so I'm using P40 GPUs to replicate/validate the pretraining pipeline.

@maxluk @aashnamsft any pretraining details that you guys can share? Even a screenshot of the AzureML metrics page would be helpful to compare and double-check.