search

NXTProduct / TUNet

52 stars 16 forks source link

Masked speech modeling #3

Closed ikou-austin closed 2 years ago

ikou-austin commented 2 years ago

Thank you for your answer to my previous question, I noticed that your paper mentions that it would be more effective to use Masked Speech Modeling for pre-training before training the BWE model, could you please provide me with the code of the MSM model and its pre-training model checkpoint file?

anhnv125 commented 2 years ago

Hi,

Unfortunately due to code refactoring and cleanup, pytorch checkpoint could not be loaded.

Pretraining the model with MSM is fairly simple as you only need to set the param task = 'msm' in the config file then execute python main.py --mode train. MSM training should be converged after around 50 epochs and then you can start bwe training from the pretrained model by specifying the argument --version (remember to modify task back to bwe).

ikou-austin commented 2 years ago

Thanks for your reply, I successfully trained the model of MSM following the steps you mentioned.