Closed freexxxyyy closed 1 year ago
We used an older version of fairseq where max_tokens
have higher priorities. So you can ignore max-sentences
.
Does it mean that the latest fairseq will always first use "max-sentences" as batch size, and ignore "max-tokens"?
Yes.
Hi,
In the pretraining script, you set the max-sentences to 32. Max-sentences is per GPU, so PER_GPU_TRAIN_BATCH_SIZE is 32. But the "max-tokens" is 2048 and the "tokens-per-sample" is 512, , so PER_GPU_TRAIN_BATCH_SIZE is 4. Why are these two parameters conflicted?
Thanks