XiangLi1999 / Diffusion-LM

Diffusion-LM
Apache License 2.0
1.02k stars 133 forks source link

The effect of "logp_term" #27

Open lgs00 opened 1 year ago

lgs00 commented 1 year ago

Thanks for sharing the code. But I would like to ask some questions. In the line of 51-54 colomn, infill_util.py. what's the effect of logp_term and how to choose the value ofcoef. Why add the logp_term to model_out.loss.
coef = 0.01 logp_term = coef * ((mean - input_embs_param) ** 2 / sigma).mean(dim=0).sum() loss = model_out.loss + logp_term

XiangLi1999 commented 1 year ago

This term helps keeping the embedding on track of producing fluent text. without this term, doing multiple step updates in model_out.loss might produce sentences that are not fluent/grammatical.

It's mentioned in the paper Section 5.1