search

locuslab / tofu

Landing Page for TOFU
MIT License
98 stars 24 forks source link

Bug in calculating loss when using DPO #30

Closed zeta-zl closed 7 months ago

zeta-zl commented 7 months ago

I have encountered what appears to be a bug in the calculation of the loss function within the dataloader.py file, specifically between lines 167 and 174. This issue arises while using the dpo method.

pi_logratios = idk_loss_current - forget_loss_current
ref_logratios = idk_loss_oracle - forget_loss_oracle

beta = 0.1
loss = -F.logsigmoid(beta * (pi_logratios - ref_logratios)).mean()
print(loss.item())
loss = -pi_logratios.mean()
loss = -idk_loss_current.mean()

It appears that the final calculation of loss as -idk_loss_current.mean() contradicts the expected retult described in the paper. Thank you for your attention to this matter.

molereddy commented 7 months ago

It was clarified on an earlier issue that the "dpo" loss is deprecated and isn't used in the results https://github.com/locuslab/tofu/issues/20#issuecomment-2027753086