what is the synthetic self-training

[lxylxyoo]

I noticed that the best model for squad2.0 uses bert + synthetic self-training trick, but i can't find any description about this. Could anyone show me how this trick work? Thanks!

[Vibha111094]

I have also been wondering about the same,I am not able to find any further details about these methods can be used along with BERT . Any details on this will be very helpful.

[ZhuoranLyu]

same question

[f-dx]

Here is the answer, from page 26

https://nlp.stanford.edu/seminar/details/jdevlin.pdf?fbclid=IwAR2TBFCJOeZ9cGhxB-z5cJJ17vHN4W25oWsjI8NqJoTEmlYIYEKG7oh4tlY

[lxylxyoo]

Here is the answer, from page 26

https://nlp.stanford.edu/seminar/details/jdevlin.pdf?fbclid=IwAR2TBFCJOeZ9cGhxB-z5cJJ17vHN4W25oWsjI8NqJoTEmlYIYEKG7oh4tlY

Thank you so much for your share !!!

[ecchochan]

So how about N-Gram Masking? Does anyone know? Even Single Model beats the others lol

[lxylxyoo]

So how about N-Gram Masking? Does anyone know? Even Single Model beats the others lol

Baidu recently open sourced a model called ERNIE, which masked n-gram instead of 1-gram during pre-training. I think this is similiar to the bert + N-Gram Masking. https://github.com/PaddlePaddle/LARK/tree/develop/ERNIE

[Dogy06]

Is synthetic self training in this repo?

[hsm207]

So how about N-Gram Masking? Does anyone know? Even Single Model beats the others lol

Baidu recently open sourced a model called ERNIE, which masked n-gram instead of 1-gram during pre-training. I think this is similiar to the bert + N-Gram Masking. https://github.com/PaddlePaddle/LARK/tree/develop/ERNIE

What is N-Gram Masking? Anyone care to translate the readme in Baidu's repo to English?