search

microsoft / MASS

MASS: Masked Sequence to Sequence Pre-training for Language Generation
https://arxiv.org/pdf/1905.02450.pdf
Other
1.12k stars 206 forks source link

How to reload checkpoint for UNMT? #154

Closed him-mah10 closed 4 years ago

him-mah10 commented 4 years ago

Hello, I am Pre-training a model for UNMT task as per the instructions given. Could you please tell me how to load a checkpoint? For XLM we could simply use --reload_checkpoint, could you tell me how to do the same here? Apparently --reload_checkpoint doesn't work. TIA

masonreznov commented 4 years ago

Umm! You do not have to mention it explicitly if you didn't change the dumped path. The following line should handle the checkpoint reloading. https://github.com/microsoft/MASS/blob/48c119dfc6d3652b9550836d114cdaff1716ff29/MASS-unsupNMT/src/trainer.py#L458

him-mah10 commented 4 years ago

@masonreznov yep, it does handle checkpoint reloading. Thank-you :) Anyone who faces the same issue, change the checkpoint_path variable manually to point to checkpoint path.

Nanamumuhan commented 4 years ago

I reloaded checkpoint.pth, but there is such an error, how to solve? Error message： Traceback (most recent call last): File "train.py", line 351, in main(params) File "train.py", line 266, in main trainer = EncDecTrainer(encoder, decoder, data, params) File "/home/ubuntu/Documents/MASS/MASS-unsupNMT/src/trainer.py", line 736, in init super().init(data, params) File "/home/ubuntu/Documents/MASS/MASS-unsupNMT/src/trainer.py", line 100, in init self.reload_checkpoint() File "/home/ubuntu/Documents/MASS/MASS-unsupNMT/src/trainer.py", line 469, in reload_checkpoint getattr(self, name).load_state_dict(data[name]) File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 847, in load_state_dict self.class.name, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for TransformerModel: Unexpected key(s) in state_dict: "attentions.2.q_lin.weight", "attentions.2.q_lin.bias", "attentions.2.k_lin.weight", "attentions.2.k_lin.bias", "attentions.2.v_lin.weight", "attentions.2.v_lin.bias", "attentions.2.out_lin.weight", "attentions.2.out_lin.bias", "attentions.3.q_lin.weight", "attentions.3.q_lin.bias", "attentions.3.k_lin.weight", "attentions.3.k_lin.bias", "attentions.3.v_lin.weight", "attentions.3.v_lin.bias", "attentions.3.out_lin.weight", "attentions.3.out_lin.bias", "layer_norm1.2.weight", "layer_norm1.2.bias", "layer_norm1.3.weight", "layer_norm1.3.bias", "ffns.2.lin1.weight", "ffns.2.lin1.bias", "ffns.2.lin2.weight", "ffns.2.lin2.bias", "ffns.3.lin1.weight", "ffns.3.lin1.bias", "ffns.3.lin2.weight", "ffns.3.lin2.bias", "layer_norm2.2.weight", "layer_norm2.2.bias", "layer_norm2.3.weight", "layer_norm2.3.bias".

him-mah10 commented 4 years ago

@Nanamumuhan did you try to load the pretrianed english model pushed by the author with the Mass-unsupNMT scripts? I am afraid it won't work. I tried a lot but couldn't get it running, if you or someone is able to do so, please let me know. TIA

StillKeepTry commented 4 years ago

@him-mah10 What is your script? You can try model from MASS-summarization if you mean English model.

him-mah10 commented 4 years ago

@StillKeepTry I wanted to know if I could use the mass-base-uncased model with the script for MASS-unsupNMT i.e. if this line points to mass-base-uncased model, will I be able to use it?

Nanamumuhan commented 4 years ago

@him-mah10 I tried many times and it didn't work. Why is this?

Nanamumuhan commented 4 years ago

@him-mah10 You can change this line to buy the absolute path to your own checkpoint.pth's, or a relative path, so you can reload checkpoint. like this ( checkpoint_path = '/home/ubuntu/Documents/MASS/MASS-unsupNMT/dumped/checkpoint.pth'). The previous error was because I changed the size of the model when I reloaded checkpoint.pth.