Closed phikoehn closed 3 years ago
Hi,
try --shuffle batches
instead of --no-shuffle
. --no-shuffle
in combination with a default setting for --maxi-batch-sort trg
probably results in your batches being sorted by increasing length inside a maxi-batch chunk. I think that would explain the bouncing. With --shuffle batches
the data gets read in in streaming fashion but the accumulated batches get shuffled within a maxi-batch. If you want to keep true input order you would need --no-shuffle
(or --shuffle none
) and --maxi-batch-sort none
but that can result in badly padded batches.
Great - this works now!
Bug description
I am mainly interested in streaming in the training data (-t stdin) to reduce RAM usage due to very large training files. However, training fails - after a while loss bounces around and BLEU scores roller-coast from 25 to 0 and anything inbetween.
How to reproduce
Regular training:
Log:
Streaming training:
Log:
BLEU:
Context