Open PeterZhizhin opened 4 months ago
Hey @PeterZhizhin feel free to close this issue, the +1 is not a bug because it is used only in target when you load a first batch and in the next batch it'll be part of the input and not target, so it's actually fine.
First, we read
B*T+1
tokens, but advance the iterator byB*T
tokens instead.Then, there is this if statement:
Possibly, we should remove the
loader->num_processes
multiplication here.We need to verify that this is the way these functions should work.