google-research / multilingual-t5

Apache License 2.0
1.25k stars 129 forks source link

Add colab notebook for fine-tuning mT5 on XNLI #44

Open mayhewsw opened 3 years ago

mayhewsw commented 3 years ago

This is basically copying the t5-trivia notebook from the T5 repo.

I've run this with mT5-small for a small number of steps (5700), and got the following validation accuracies.

INFO:absl:eval/xnli_dev_test.all_langs/accuracy at step 1005700: 65.673
INFO:absl:eval/xnli_dev_test.ar/accuracy at step 1005700: 62.610
INFO:absl:eval/xnli_dev_test.bg/accuracy at step 1005700: 68.956
INFO:absl:eval/xnli_dev_test.de/accuracy at step 1005700: 68.072
INFO:absl:eval/xnli_dev_test.el/accuracy at step 1005700: 67.671
INFO:absl:eval/xnli_dev_test.en/accuracy at step 1005700: 79.960
INFO:absl:eval/xnli_dev_test.es/accuracy at step 1005700: 72.129
INFO:absl:eval/xnli_dev_test.fr/accuracy at step 1005700: 69.157
INFO:absl:eval/xnli_dev_test.hi/accuracy at step 1005700: 61.446
INFO:absl:eval/xnli_dev_test.ru/accuracy at step 1005700: 67.309
INFO:absl:eval/xnli_dev_test.sw/accuracy at step 1005700: 55.422
INFO:absl:eval/xnli_dev_test.th/accuracy at step 1005700: 64.177
INFO:absl:eval/xnli_dev_test.tr/accuracy at step 1005700: 61.767
INFO:absl:eval/xnli_dev_test.ur/accuracy at step 1005700: 57.912
INFO:absl:eval/xnli_dev_test.vi/accuracy at step 1005700: 63.333
INFO:absl:eval/xnli_dev_test.zh/accuracy at step 1005700: 65.181
google-cla[bot] commented 3 years ago

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.


What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

mayhewsw commented 3 years ago

@googlebot I signed it!

lintingxue commented 2 years ago

Hey Stephen, thanks for the suggestion, unfortunately we don't accept external PRs for this repo.