search

bigcode-project / starcoder

Home of StarCoder: fine-tuning & inference!
Apache License 2.0
7.33k stars 522 forks source link

Why utilizing the 'question' column of the 'ArmelR/stack-exchange-instruction' dataset for gradient backpropagation? #135

Open HIT-cwh opened 1 year ago

HIT-cwh commented 1 year ago

Thank you for your valuable open-source contribution!

In instruction tuning stage, it seems that only the answer aligned with the instruction participates in the backpropagation process. And this code seems to imply that the question part of the dataset is also involved in backpropagation of the gradient. Will this lead to better training results?

image

ArmelRandy commented 1 year ago

Hi. Thank you for reaching out. In this implementation, we do not mask the part of the sequence which corresponds to the question. You can slightly modify the code to account for that. Both methods work reasonably well (masking and not masking) though masking seems to be the standard practice.