How to pretrain about other language

[wanaDoctor]

Hello, thanks for sharing this awesome paper. I want to pretrain this model in Korean. But this project is not provide pretraining code. if you don't mind me asking, would you provide pretraining code?

thanks :)

[qiweizhen]

Multi-language pretrained model will be released in near future.

Pretraining code is not schedulred to release. But you can simply use Fairseq build-in pretraining tasks or format your data in input-output format to pretrain a Kerean model.

[nickcom007]

Hello, thanks for sharing this awesome paper. I want to pretrain this model in Korean. But this project is not provide pretraining code. if you don't mind me asking, would you provide pretraining code?

thanks :)

Hi, have you successfully pre-train in another language?

[yuyan2do]

@nickcom007 See cross-lingual version at this page

[rkoyanagi]

Thank you for sharing great code. I also want to pretrain ProphetNet in other language and trying, but I cound not success even in English yet. (no cross-lingual, because I want to use only our vocabulary of our language and domainf without noise.)

But you can simply use Fairseq build-in pretraining tasks

Does it means fairseq-train --task masked_lm or another one? Could you show me a sample command to reproduct?

Or

format your data in input-output format to pretrain a Kerean model.

Could you show me an example of input-output format of pretrain (in English)?

[rkoyanagi]

Using the mass_s2s task from MASS , it worked.

[ghost]

@rkoyanagi Can you show me how to train new model for other languages? I am using it for Vietnamese Summarization. Thanks.