Criteria to increase input length

[sohviluukkonen]

Hi! In your paper shortly mention the procedure of increasing the sequence length once the loss plateaus:

Initially, we used inputs of length L = 2^11 tokens with a batch size of 64. We doubled input length each time the loss reached a plateau, simultaneously reducing batch size to maintain a fixed total number of tokens per batch.

Unfortunately, I could not find either in the paper or the code, how do you define a plateau? Could you give me some information on this?

Thanks!

[damiano-sg]

Hi, that was defined empirically. Practically we increased L every time we observed a significative change in the slope of the loss. We didn't follow a precise schedule for the input length.

[sohviluukkonen]

Thank you for your quick answer!