search

timoschick / pet

This repository contains the code for "Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference"
https://arxiv.org/abs/2001.07676
Apache License 2.0
1.62k stars 283 forks source link

LM training for patterns #53

Closed yujianll closed 2 years ago

yujianll commented 2 years ago

Hi,

It seems the auxiliary LM training includes the patterns. i.e. words in patterns also have the possibility of being masked. I wonder if there is any reason that you include the patterns in LM training, instead of using the original dataset. Many thanks.

timoschick commented 2 years ago

Hi @yujianll, the original idea was to get the language model accustomed to the used patterns. We did not run any experiments where the patterns are not used for LM training, but I would assume that this plays at most a very minor role for final model performance.

yujianll commented 2 years ago

Thanks! This makes sense.