search

tianyi-lab / Cherry_LLM

[NAACL'24] Self-data filtering of LLM instruction-tuning data using a novel perplexity-based difficulty score, without using any other models
287 stars 19 forks source link

Logic behind IFD score #4

Closed mshen2 closed 11 months ago

mshen2 commented 11 months ago

Thanks for your work! I am currently trying to understand the logic behind IFD score and maybe I am misunderstanding something. For equation 4 of the direct answer score, the paper mentions "A higher direct answer score may suggest that the answer is inherently more challenging or intricate for the model to generate". However, isn't it when equation 4 (the probability of the sentence) gives a higher value indicates that the sentence makes more sense so it would be more natural for LM to generate? kindly looking forward to your reply. Thanks!

MingLiiii commented 11 months ago

Thanks for asking! We are really sorry for the confusion. There is a typo in Equation 3 and Equation 4. There should be minus signs on these two equations.

Apologize again for the mistakes, we will correct this typo and update our manuscript in the next version. Let me know if you still have problems~