search

yegcjs / DINOISER

25 stars 3 forks source link

Training Unsuccessful: Code Replication for IWSLT14 DE->EN #1

Open chiral-carbon opened 6 months ago

chiral-carbon commented 6 months ago

Hi,

Thanks for sharing your interesting work! I replicated this codebase without changing any hyperparams for the IWSLT14 German to English translation task and my model stopped training after 1368 epochs. On evaluation however the reported BLEU is only 3.77. 

BLEU = 3.77 9.3/5.0/2.8/1.6 (BP = 1.000 ratio = 7.597 hyp_len = 973814 ref_len = 128189)

Would you have any insights on why this might have happened? My training stopped on its own presumably due to the early stopping criteria as the losses started stagnating. You will notice that the hypothesis and reference sentence length ratio is also very high and it gives a lot of filler words like punctuation tokens and repeated words to fill it.

Would appreciate any insight you might have as to why this might have happened. Thanks!

yegcjs commented 6 months ago

Thanks for your attention to our work.

Regarding the problem you encounter, it is probably issues relating the length predictor. Our training scripts involve two steps. We first train the diffusion language models with oracle target length. After this, you should use the --oracle-length option for sampling. The second step trains a length predictor, after which you can sample with the predicted target lengths.