search

peterwestuw / surface-form-competition

58 stars 9 forks source link

Questions about Template Design #3

Closed bflashcp3f closed 3 years ago

bflashcp3f commented 3 years ago

Hi,

Thank you for this interesting work, and congratulations on the EMNLP acceptance! After reading the paper, one question I am having is that how those domain-specific templates are designed and selected. In the get_sst2_variant_template function (data_loaders.py), I see a couple of template candidates, so I guess what you did is to first manually design a couple of template candidates for each domain/task and then select the best one based on the dev set performance (if both dev and test sets exist), right? Please correct me if I am wrong and look forward to your reply. Thanks!

ari-holtzman commented 3 years ago

Hi,

Great question! That is indeed what we did, though the variants in the sst-2 template are from our template robustness experiments (which aren't on Arxiv yet, but will be very soon); for that, we used the templates from "Calibrate Before Use" https://arxiv.org/abs/2102.09690