Open lgs00 opened 1 year ago
This term helps keeping the embedding on track of producing fluent text. without this term, doing multiple step updates in model_out.loss might produce sentences that are not fluent/grammatical.
It's mentioned in the paper Section 5.1
Thanks for sharing the code. But I would like to ask some questions. In the line of 51-54 colomn, infill_util.py. what's the effect of
logp_term
and how to choose the value ofcoef
. Why add thelogp_term
to model_out.loss.coef = 0.01
logp_term = coef * ((mean - input_embs_param) ** 2 / sigma).mean(dim=0).sum()
loss = model_out.loss + logp_term