Would you like to release the code about how to train a bigbird with other language

google-research / bigbird

Transformers for Longer Sequences

https://arxiv.org/abs/2007.14062

Apache License 2.0

563 stars 101 forks source link

Would you like to release the code about how to train a bigbird with other language #5

Closed RyanHuangNLP closed 3 years ago

RyanHuangNLP commented 3 years ago

@manzilz I want to train a bigbird with other language

manzilz commented 3 years ago

Hi Ryan

Can you be more specific in what is challenge you are facing in training bigbird with other languages? For example if you want to train a masked LM, we have provided the pretraining code in bigbird/pretrain/. You should be able to use it to train for any language. You might want to use your vocabulary specific to the language.

Thanks!