FInal token index when training a model

[robertodessi]

Hi, thanks a lot for making the code available, it's a great resource to use!

I was wondering why the index when computing the loss from the output of gpt is shifted by 1 on the left: https://github.com/rmokady/CLIP_prefix_caption/blob/main/train.py#L315

shouldn't it be logits = outputs.logits[:, dataset.prefix_length :]

Thanks!

[Twilighter9527]

when I follow you change logits to be "logits = outputs.logits[:, dataset.prefix_length :]",the loss become very slow quickly and the inference caption become "\n\n\n\n.......", do you have this situation？or that should be "logits = outputs.logits[:, dataset.prefix_length - 1: -1]"thank you.

[baaaad]

the gpt2 acts the ''next token predixtion', so the token in time step t is predicted by the token in time step t-1. the final token does not need to predict for the next token. so it is right to use logits = outputs.logits[:, dataset.prefix_length -1 : -1]

[Twilighter9527]

you can see the source

[robertodessi]

True, makes sense thanks!