search

browsermt / students

Efficient teacher-student models and scripts to make them
Other
48 stars 19 forks source link

Start en-pl and en-fr training #62

Open kpu opened 2 years ago

kpu commented 2 years ago

These can get a leg up by using the existing pl-en (#61) and fr-en (#60) models that have already built their teacher systems, which should then be used for back translation. Note that forward translation for making a pl-en student is the same thing as backtranslation for making an en-pl model.

See https://github.com/mozilla/firefox-translations-training/issues/46 on how to short-circuit this and talk to @eu9ene as necessary.

ugermann commented 2 years ago

Note that forward translation for making a pl-en student is the same thing as backtranslation for making an en-pl model.

Is it? Doesn't forward translation include the parallel data and backtranslation is typically only performed on monolingual data in the target language? In practise it may not make much of a difference, but I'm not sure they're exactly the same.

eu9ene commented 2 years ago

Hi, what's the status of the en-fr model training? At what stage in the pipeline is it now?

ugermann commented 2 years ago

Currently training first teacher fr-en, en-fr not startet yet (hoping to use the first teacher for backtranslation). There were numerous issues with the pipeline (party due to things getting out of sync between different branches and repositories) which sent me to jail without passing start go or collecting $200. We have over 600M sentence pairs for teacher training, so I expect things to be slow (first attempt went haywire, probably because the learning rate was too high). I've always been but am increasingly even more sceptical about the feasibility of a one-size-fits-all training procedure that works like a breadmaker (through in the ingredients, press a button and wake up the next morning to freshly baked bread / MT models). It's more like intensive care: needs constant monitoring of the patient and adjustments in response to signal readings.

2022-04-01 12:36:42] [valid] Ep. 1 : Up. 3000 : bleu-detok : 0.726982 : new best
[2022-04-01 14:29:42] [valid] Ep. 1 : Up. 6000 : bleu-detok : 0 : stalled 1 times (last best: 0.726982)
[2022-04-01 16:22:25] [valid] Ep. 1 : Up. 9000 : bleu-detok : 0 : stalled 2 times (last best: 0.726982)
[2022-04-01 18:13:56] [valid] Ep. 1 : Up. 12000 : bleu-detok : 1.3157 : new best
[2022-04-01 20:06:32] [valid] Ep. 1 : Up. 15000 : bleu-detok : 1.00074 : stalled 1 times (last best: 1.3157)
[2022-04-01 21:58:15] [valid] Ep. 1 : Up. 18000 : bleu-detok : 2.54976 : new best
[2022-04-01 23:47:29] [valid] Ep. 1 : Up. 21000 : bleu-detok : 3.16497 : new best