Open EtienneAb3d opened 1 year ago
Remark: previous post's graph updated for a better comparison with the one below.
I don't have absolute certainty, but I may have an explanation, and a solution. It can also be of interest for common bilingual translation tasks.
In this mono-lingual denoising task, target sentences may be present in many copies in the training corpus:
By default, Marian is sorting maxi-batches according to target sentences, thus maxi-batch-sort trg
. This may cause several mini-batches to be filled with a single same target multi-occurring sentence, or a very few number of different sentences. I think this is producing over-fitting on these mini-batches, destroying the whole model quality in a very short time.
To solve this, ideally, sentences should be shuffled without any sorting, thus with option maxi-batch-sort none
. But, this is causing a 3x slower training. Quite prohibitive.
A half-solution is to use maxi-batch-sort src
. As same target sentences have different source noisy sentences, it will ensure a minimum diversity in each mini-batch, avoiding too much identical target sentences to be grouped together.
Doing this (with the same Seed, thus the same sentence shuffled order), I get a little faster training, without brutally decreasing BLEU:
An unrelated question - what do you use to generate the graphs?
Hi!
What I'm doing is a bit particular: translating noisy text to denoised clean text (for NeuroSpell auto-corrector). On the principle, it should not cause the strange thing described below. It works perfectly with ModernMT for several years. I'm now experimenting with Marian.
I'm using:
On one training (not all), the BLEU is brutally decreasing for several batches, while the Cost is increasing in the same kind of way. It then increases again.
If the data are shuffled, I do not see a reason why a lot of bad sentence pairs would be concentrated on a sequence of several batches. And, the first Epoch is OK. This occurs only on the second Epoch (and certainly also on the third one).
This is reproducible using the same Seed parameter (still not tested with an other value).
Did you see a reason why this could occur?
Training curves (due to the task achieved, it's normal to get a high BLEU):![image](https://user-images.githubusercontent.com/25932245/200236561-5fa63b82-e67c-4705-a89b-86c174e36f0b.png)
PS: updated graph