I'm trying to pre-train a language model with 8 monolingual data using MASS-unsup. I have individually learned and applied BPE to the monolingual data and ran the preprocess.py script successfully.
I am encountering an error whenever i'm trying to run the following training snippet
I'm trying to pre-train a language model with 8 monolingual data using MASS-unsup. I have individually learned and applied BPE to the monolingual data and ran the
preprocess.py
script successfully.I am encountering an error whenever i'm trying to run the following training snippet
`python train.py \
--exp_name pretrain-fl \ --data_path data/processed/ \ --mass_steps 'l1,l2,l3,l4,l5,l6,l7,l8' \ --encoder_only false \ --emb_dim 1024 \ --n_layers 6 \ --n_heads 8 \ --dropout 0.1 \ --attention_dropout 0.1 \ --gelu_activation true \ --tokens_per_batch 3000 \ --optimizer adam_inverse_sqrt,beta1=0.9,beta2=0.98,lr=0.0001 \ --epoch_size 200000 \ --max_epoch 100 \ --eval_bleu true \ --word_mass 0.5 \ --min_len 5 `
I'm encountering the following error:
Did anyone encounter the same error?