Closed lucasnewman closed 1 year ago
dude! you MVP - i was about to do all this this weekend
caught up with Voicebox work today and tomorrow, but will circle back to this on Sunday and get it merged! 🙏 (looks great on a cursory scan)
:100: :rocket:
even humored me and followed inline with the strange styling :laughing:
I was able to get both the pretraining and backtranslation tasks working using graphemes, as described in Appendix G in the paper. It required some model tweaks to allow freezing the encoder & specifying a target mask when the modalities are different. I added beam search as optional in the generation loop since it seems to outperform temperature sampling for the backtranslation task.
I also fixed an issue with loading checkpoints and to allow restoring checkpoints without the optimizer state. This makes both resuming training runs and fine-tuning from the pretrained model possible.
This might be too big of a change to take, but I thought it might be useful for others! Let me know if you'd prefer me to break it up into smaller changes if that's helpful.
Here's an example of a sample generation using the (severely undertrained 🙈, in my case) models: