search

tianyi-lab / Cherry_LLM

[NAACL'24] Self-data filtering of LLM instruction-tuning data using a novel perplexity-based difficulty score, without using any other models
287 stars 19 forks source link

Question about the effect of labels[0, :start_token] = -100 #20

Closed lygjwy closed 2 months ago

lygjwy commented 3 months ago

Hi, I don't understand the function of code 'labels[0, :start_token] = -100' in Line 68 from cherry_seletion/data_analysis.py file.

MingLiiii commented 2 months ago

Sorry for the late reply! The use of = -100 is to make LLMs ignore the calculation of loss on those positions.