Closed yujianll closed 2 years ago
Hi @yujianll, the original idea was to get the language model accustomed to the used patterns. We did not run any experiments where the patterns are not used for LM training, but I would assume that this plays at most a very minor role for final model performance.
Thanks! This makes sense.
Hi,
It seems the auxiliary LM training includes the patterns. i.e. words in patterns also have the possibility of being masked. I wonder if there is any reason that you include the patterns in LM training, instead of using the original dataset. Many thanks.