Question (not issue) related to dataset generation

[insop]

Hi @loubnabnl

Thank you so much for this nice repo for running finetuning.

I have one question and did not find a better way to communicate, so feel free to answer and then close this issue.

In the following code, input_ids and labels are the same for supervised fine tuning. Is there somewhere in the model training parameter that knows it is a causal LM training, so it will shift the labels by one, so that input_ids and labels become a next token prediction task?

• https://github.com/loubnabnl/santacoder-finetuning/blob/13b69333fe60d0201c35f40ca0c5a8bfce104f80/train.py#L175

...
            for example in examples:
                self.current_size += 1
                yield {
                        "input_ids": torch.LongTensor(example),
                        "labels": torch.LongTensor(example),
                    }

[loubnabnl]

Yes the labels are shifted in transformers here

[insop]

Yes the labels are shifted in transformers here

Thank you @loubnabnl !