Confused about the "max-sentences" in pretraining

[freexxxyyy]

Hi,

In the pretraining script, you set the max-sentences to 32. Max-sentences is per GPU, so PER_GPU_TRAIN_BATCH_SIZE is 32. But the "max-tokens" is 2048 and the "tokens-per-sample" is 512, , so PER_GPU_TRAIN_BATCH_SIZE is 4. Why are these two parameters conflicted?

Thanks

[wasiahmad]

We used an older version of fairseq where max_tokens have higher priorities. So you can ignore max-sentences.

[freexxxyyy]

Does it mean that the latest fairseq will always first use "max-sentences" as batch size, and ignore "max-tokens"?

[wasiahmad]

Yes.