The calculation of number of epochs are incorrect

[westn]

First, Thanks for releasing the code and being as open as possible with it.

I noted that the epoch logging seems to be based on sys.maxsize instead of the actual size of the current dataset. Is this a low-hanging fruit to fix?

https://github.com/facebookresearch/MobileLLM/blob/1e67bf6a831b0d3863217f6a1dfd187919f636ac/pretrain.py#L91-L93

This also affects if you use the parameter --num_train_epochs 1 for the script.

[zxdmike]

This is set by design. The dataloader doesn't preload all the data as the data file could be huge; it continuously streams data, so the total length per epoch is unknown. Instead of using --num_train_epochs to control training duration, we use --max_steps.