XLM-R: finetune command similar to roberta?

[cindyxinyiwang]

I want to use the XLM-R model to finetune on the XNLI dataset. If I understand correctly, the finetuning command is basically the same with roberta(https://github.com/pytorch/fairseq/tree/master/examples/roberta)? Can I basically use the roberta finetune command, replacing the roberta model/vocab with xlm-r model/vocab?

[ngoyal2707]

@cindyxinyiwang The commands are mostly same but the preprocessing steps are slightly different. As we use Sentence Piece Model in XLM-R instead of GPT2-BPE codes.

It's on my list to write the detailed finetuning cmds, it might take 1-2 weeks. Feel free to share your cmds and I can help you.

[cindyxinyiwang]

Could you take a look at my preprocessing script below? I could use the processed data to finetune the xlmr model, using --arch roberta_large. The finetuning would run for several hundred steps, then throw an error that says tensor src index out of bound.

MODEL=pretrained_models/xlmr.large/sentencepiece.bpe.model DICT=pretrained_models/xlmr.large/dict.txt

for file in xnli/data/.input0; do f=${file/input0/input0.spm} spm_encode --model=$MODEL --output_format=piece < $file > $f done

for file in xnli/data/.input1; do f=${file/input1/input1.spm} spm_encode --model=$MODEL --output_format=piece < $file > $f done

for file in xnli/data/.label; do f=${file/label/label.spm} spm_encode --model=$MODEL --output_format=piece < $file > $f done

for LAN in ar bg de el en es fr hi ru sw th tr ur vi zh; do for type in input0 input1; do python preprocess.py \ --only-source \ --trainpref xnli/data/xnli.train."$type".spm."$LAN" \ --validpref xnli/data/xnli.dev."$type".spm."$LAN" \ --testpref xnli/data/xnli.test."$type".spm."$LAN" \ --destdir data-bin/xnli/"$LAN"/"$type" \ --workers 32 \ --srcdict $DICT done done python preprocess.py \ --only-source \ --trainpref xnli/data/xnli.train.label.ar \ --validpref xnli/data/xnli.dev.label.ar \ --testpref xnli/data/xnli.test.label.ar \ --destdir data-bin/xnli/ar/label \ --workers 32

for LAN in bg de el en es fr hi ru sw th tr ur vi zh; do python preprocess.py \ --only-source \ --trainpref xnli/data/xnli.train.label."$LAN" \ --validpref xnli/data/xnli.dev.label."$LAN" \ --testpref xnli/data/xnli.test.label."$LAN" \ --destdir data-bin/xnli/"$LAN"/label \ --srcdict data-bin/xnli/ar/label/dict.txt \ --workers 32 done

[yunshenzhou]

@ngoyal2707 Hello, Currently I'm working with a project which applies XLM-R. Could you please upload the finetuning cmds when you are convenient? It is very helpful for our users to check whether we apply the XLM-R properly. Thank you so much.

[ngoyal2707]

Hey @as3071 , I will try to get to XNLI cmd soon. In the meantime, could you please refer to pytext repo?

[yunshenzhou]

Hi @ngoyal2707 I've tried to open the pytext repo. In the documentation, it says

Tutorial in Notebook https://github.com/facebookresearch/pytext/blob/master/demo/notebooks/xlm_r_tutorial.ipynb

Run the tutorial in Google Colab https://colab.research.google.com/github/facebookresearch/pytext/blob/master/demo/notebooks/xlm_r_tutorial.ipynb

However, I tried to open the links, neither of them works. Can you check what happens when you are convenient? Thank you so much.

[swethathomas]

Hi @ngoyal2707 I'm trying to fine tune XLM-R on the XNLI dataset. Would it be possible for you to upload the fine-tuning cmds when convenient this would be very helpful! Thank you so much.

[YeWenting]

Hi @ngoyal2707 I'm trying to fine tune XLM-R on my own dataset. Your work is impressive and I am very appreciate your help! Thank you so much!

[yuchenlin]

Hey @as3071 , I will try to get to XNLI cmd soon. In the meantime, could you please refer to pytext repo?

Hi @ngoyal2707 , I was wondering where we can find the example command to reproduce the XLM-R's results on XNLI. Would you please point it out if it is already in the repo?

[stale[bot]]

This issue has been automatically marked as stale. If this issue is still affecting you, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[stale[bot]]

Closing this issue after a prolonged period of inactivity. If this issue is still present in the latest release, please create a new issue with up-to-date information. Thank you!