search

XiangLi1999 / ContrastiveDecoding

contrastive decoding
173 stars 11 forks source link

Question about ignore_prefix flag #2

Closed martiansideofthemoon closed 1 year ago

martiansideofthemoon commented 1 year ago

Hi @XiangLi1999 and @ari-holtzman, Thanks for releasing your codebase and interesting work! I had a quick question about the --ignore_prefix flag. Would you suggest setting this to yes or no? In the ablation experiments in Table 5, it seems like setting it to yes was slightly better? However, the default setting in the README file is no.

I was wondering how much this setting affects MAUVE score and quality, and what would you suggest?

Best Wishes, Kalpesh

XiangLi1999 commented 1 year ago

Hi Kalpesh,

Thanks for the question! all the main results in the ppaer uses --ignore_prefix yes (except for the ablation). This setting affect the coherence score, but not very much the MAUVE score. For some domains, I prefer the samples of --ignore_prefix no, but overall, they are similar.

martiansideofthemoon commented 1 year ago

Great, thank you @XiangLi1999!