microsoft / MASS

MASS: Masked Sequence to Sequence Pre-training for Language Generation
https://arxiv.org/pdf/1905.02450.pdf
Other
1.11k stars 206 forks source link

error in running training script for pre-training multiple monolingual data #146

Closed renziver closed 4 years ago

renziver commented 4 years ago

I'm trying to pre-train a language model with 8 monolingual data using MASS-unsup. I have individually learned and applied BPE to the monolingual data and ran the preprocess.py script successfully.

I am encountering an error whenever i'm trying to run the following training snippet

`python train.py \

--exp_name pretrain-fl \ --data_path data/processed/ \ --mass_steps 'l1,l2,l3,l4,l5,l6,l7,l8' \ --encoder_only false \ --emb_dim 1024 \ --n_layers 6 \ --n_heads 8 \ --dropout 0.1 \ --attention_dropout 0.1 \ --gelu_activation true \ --tokens_per_batch 3000 \ --optimizer adam_inverse_sqrt,beta1=0.9,beta2=0.98,lr=0.0001 \ --epoch_size 200000 \ --max_epoch 100 \ --eval_bleu true \ --word_mass 0.5 \ --min_len 5 `

I'm encountering the following error:

Screen Shot 2020-06-01 at 8 36 01 PM

Did anyone encounter the same error?