search

facebookresearch / ParlAI

A framework for training and evaluating AI models on a variety of openly available dialogue datasets.
https://parl.ai
MIT License
10.49k stars 2.1k forks source link

ValueError: ParlaiDialogTeacher requires a "text" field in every entry, but one is missing in Line 1 in /home/weichen/ParlAI/data/blended_skill_talk/train.txt. #2640

Closed Dogy06 closed 4 years ago

Dogy06 commented 4 years ago

Bug description Please enter a clear and concise description of what the bug is. ValueError: ParlaiDialogTeacher requires a "text" field in every entry, but one is missing in Line 1 in /home/weichen/ParlAI/data/blended_skill_talk/train.txt. when running the training code for 90M Blender model finetuning. Reproduction steps Enter steps to reproduce the behavior. setup ParlAI, run: python examples/train_model.py -t blended_skill_talk,wizard_of_wikipedia,convai2:normalized -m transformer/generator --multitask-weights 1,3,3,3 --init-model zoo:tutorial_transformer_generator/model --dict-file zoo:tutorial_transformer_generator/model.dict --embedding-size 512 --n-layers 8 --ffn-size 2048 --dropout 0.1 --n-heads 16 --learn-positional-embeddings True --n-positions 512 --variant xlm --activation gelu --skip-generation True --fp16 True --text-truncate 512 --label-truncate 128 --dict-tokenizer bpe --dict-lower True -lr 1e-06 --optimizer adamax --lr-scheduler reduceonplateau --gradient-clip 0.1 -veps 0.25 --betas 0.9,0.999 --update-freq 1 --attention-dropout 0.0 --relu-dropout 0.0 --skip-generation True -vp 15 -stim 60 -vme 20000 -bs 16 -vmt ppl -vmm min --save-after-valid True --model-file /tmp/test_train_90M

Expected behavior Give a clear and concise description of what you expected to happen. The model should have loaded the data. Logs Please paste the command line output:

(parlai_cb) weichen@polarbear:~/ParlAI$ python examples/train_model.py -t blended_skill_talk,wizard_of_wikipedia,convai2:normalized -m transformer/generator --multitask-weights 1,3,3,3 --init-model zoo:tutorial_transformer_generator/model --dict-file zoo:tutorial_transformer_generator/model.dict --embedding-size 512 --n-layers 8 --ffn-size 2048 --dropout 0.1 --n-heads 16 --learn-positional-embeddings True --n-positions 512 --variant xlm --activation gelu --skip-generation True --fp16 True --text-truncate 512 --label-truncate 128 --dict-tokenizer bpe --dict-lower True -lr 1e-06 --optimizer adamax --lr-scheduler reduceonplateau --gradient-clip 0.1 -veps 0.25 --betas 0.9,0.999 --update-freq 1 --attention-dropout 0.0 --relu-dropout 0.0 --skip-generation True -vp 15 -stim 60 -vme 20000 -bs 16 -vmt ppl -vmm min --save-after-valid True --model-file /tmp/test_train_90M [ building dictionary first... ] [ dictionary already built .] [ no model with opt yet at: /tmp/test_train_90M(.opt) ]

[ WARNING ] : your model is being loaded with opts that do not exist in the model you are initializing the weights with: dynamic_batching: None,datapath: /home/weichen/ParlAI/data,label_type: response,include_knowledge: True,include_checked_sentence: True,include_knowledge_separator: False,num_topics: 5,n_encoder_layers: -1,n_decoder_layers: -1,model_parallel: False,beam_length_penalty: 0.65,topk: 10,topp: 0.9,beam_delay: 30,beam_blacklist_filename: None,temperature: 1.0,compute_tokenized_bleu: False,interactive_mode: False,fp16_impl: apex,force_fp16_tokens: False,adafactor_eps: (1e-30, 0.001),history_add_global_end_token: None,bpe_vocab: None,bpe_merge: None,bpe_add_prefix_space: None,max_lr_steps: -1,invsqrt_lr_decay_gamma: -1,parlai_home: /home/weichen/ParlAI

[ WARNING ] : your model is being loaded with opts that differ from the model you are initializing the weights with. Add the following args to your run command to change this:

--task internal:new-reddit:presorted --download-path /private/home/roller/working/parlai/downloads --datatype train:stream --multitask-weights 1 --batchsize 48 --num-epochs 5.0 --validation-every-n-secs 1800.0 --save-every-n-secs -1 --validation-every-n-epochs -1 --validation-max-exs 9920 --short-final-eval True --validation-patience 0 --dict-build-first True --load-from-checkpoint True --numworkers 4 --pytorch-preprocess False --pytorch-teacher-batch-sort False --batch-sort-cache-type pop --batch-length-range 5 --shuffle False --batch-sort-field text --pytorch-context-length -1 --pytorch-include-labels True --log-every-n-secs 30.0 --distributed-world-size 64 --verbose False --port 61337 --beam-size 8 --beam-min-n-best 3 --beam-min-length 10 --skip-generation False --inference beam --optimizer fused-adam --learningrate 0.0005 --gradient-clip 10.0 --adam-eps 1e-06 --betas 0.9,0.98 --weight-decay 0.01 --lr-scheduler invsqrt --warmup-updates 20000 --gpu 0 --beam-block-ngram 3 --beam-context-block-ngram 3

[ Using CUDA ] /home/weichen/ParlAI/parlai/utils/fp16.py:144: UserWarning: You set --fp16 true with --fp16-impl apex, but fp16 with apex is unavailable. To use apex fp16, please install APEX from https://github.com/NVIDIA/apex. 'You set --fp16 true with --fp16-impl apex, but fp16 ' Dictionary: loading dictionary from /home/weichen/ParlAI/data/models/tutorial_transformer_generator/model.dict [ num words = 54944 ] Total parameters: 87,508,992 (87,508,992 trainable) [ Loading existing model params from /home/weichen/ParlAI/data/models/tutorial_transformer_generator/model ] WARNING: not loading optim state since optim class changed. [creating task(s): blended_skill_talk,wizard_of_wikipedia,convai2:normalized] [building data: /home/weichen/ParlAI/data/blended_skill_talk] [ downloading: http://parl.ai/downloads/blended_skill_talk/blended_skill_talk.tar.gz to /home/weichen/ParlAI/data/blended_skill_talk/blended_skill_talk.tar.gz ] Downloading blended_skill_talk.tar.gz: 0%| | 32.8k/38.1M [00:01<24:37, 25.8kB/Downloading blended_skill_talk.tar.gz: 0%| | 65.5k/38.1M [00:01<18:08, 34.9kB/Downloading blended_skill_talk.tar.gz: 0%| | 131k/38.1M [00:01<13:03, 48.4kB/sDownloading blended_skill_talk.tar.gz: 1%| | 262k/38.1M [00:01<09:23, 67.2kB/sDownloading blended_skill_talk.tar.gz: 1%| | 459k/38.1M [00:02<06:57, 90.2kB/sDownloading blended_skill_talk.tar.gz: 2%| | 885k/38.1M [00:02<04:51, 128kB/s]Downloading blended_skill_talk.tar.gz: 3%| | 1.05M/38.1M [00:02<03:38, 170kB/sDownloading blended_skill_talk.tar.gz: 3%| | 1.18M/38.1M [00:02<02:44, 225kB/sDownloading blended_skill_talk.tar.gz: 3%| | 1.31M/38.1M [00:02<02:04, 295kB/sDownloading blended_skill_talk.tar.gz: 4%| | 1.44M/38.1M [00:02<01:37, 375kB/sDownloading blended_skill_talk.tar.gz: 4%| | 1.57M/38.1M [00:03<01:17, 474kB/sDownloading blended_skill_talk.tar.gz: 4%| | 1.70M/38.1M [00:03<01:02, 582kB/sDownloading blended_skill_talk.tar.gz: 5%| | 1.90M/38.1M [00:03<00:49, 725kB/sDownloading blended_skill_talk.tar.gz: 5%| | 2.06M/38.1M [00:03<00:42, 852kB/sDownloading blended_skill_talk.tar.gz: 6%| | 2.23M/38.1M [00:03<00:37, 959kB/sDownloading blended_skill_talk.tar.gz: 6%| | 2.42M/38.1M [00:03<00:32, 1.11MB/Downloading blended_skill_talk.tar.gz: 7%| | 2.59M/38.1M [00:03<00:33, 1.07MB/Downloading blended_skill_talk.tar.gz: 7%| | 2.79M/38.1M [00:03<00:29, 1.22MB/Downloading blended_skill_talk.tar.gz: 8%| | 2.98M/38.1M [00:03<00:27, 1.29MB/Downloading blended_skill_talk.tar.gz: 8%| | 3.15M/38.1M [00:04<00:27, 1.27MB/Downloading blended_skill_talk.tar.gz: 9%| | 3.31M/38.1M [00:04<00:26, 1.29MB/Downloading blended_skill_talk.tar.gz: 9%| | 3.47M/38.1M [00:04<00:26, 1.29MB/Downloading blended_skill_talk.tar.gz: 10%| | 3.64M/38.1M [00:04<00:30, 1.15MB/Downloading blended_skill_talk.tar.gz: 10%| | 3.77M/38.1M [00:04<00:29, 1.18MB/Downloading blended_skill_talk.tar.gz: 10%|1| 3.90M/38.1M [00:04<00:33, 1.03MB/Downloading blended_skill_talk.tar.gz: 11%|1| 4.03M/38.1M [00:04<00:31, 1.08MB/Downloading blended_skill_talk.tar.gz: 11%|1| 4.19M/38.1M [00:05<00:28, 1.19MB/Downloading blended_skill_talk.tar.gz: 11%|1| 4.36M/38.1M [00:05<00:26, 1.28MB/Downloading blended_skill_talk.tar.gz: 12%|1| 4.52M/38.1M [00:05<00:25, 1.33MB/Downloading blended_skill_talk.tar.gz: 12%|1| 4.69M/38.1M [00:05<00:24, 1.36MB/Downloading blended_skill_talk.tar.gz: 13%|1| 4.85M/38.1M [00:05<00:23, 1.41MB/Downloading blended_skill_talk.tar.gz: 13%|1| 5.01M/38.1M [00:05<00:23, 1.40MB/Downloading blended_skill_talk.tar.gz: 14%|1| 5.18M/38.1M [00:05<00:22, 1.45MB/Downloading blended_skill_talk.tar.gz: 14%|1| 5.34M/38.1M [00:05<00:22, 1.47MB/Downloading blended_skill_talk.tar.gz: 14%|1| 5.51M/38.1M [00:05<00:21, 1.50MB/Downloading blended_skill_talk.tar.gz: 15%|1| 5.70M/38.1M [00:06<00:21, 1.52MB/Downloading blended_skill_talk.tar.gz: 15%|1| 5.87M/38.1M [00:06<00:21, 1.47MB/Downloading blended_skill_talk.tar.gz: 16%|1| 6.03M/38.1M [00:06<00:21, 1.50MB/Downloading blended_skill_talk.tar.gz: 16%|1| 6.23M/38.1M [00:06<00:20, 1.55MB/Downloading blended_skill_talk.tar.gz: 17%|1| 6.39M/38.1M [00:06<00:22, 1.42MB/Downloading blended_skill_talk.tar.gz: 17%|1| 6.55M/38.1M [00:06<00:22, 1.38MB/Downloading blended_skill_talk.tar.gz: 18%|1| 6.72M/38.1M [00:06<00:22, 1.40MB/Downloading blended_skill_talk.tar.gz: 18%|1| 6.91M/38.1M [00:06<00:20, 1.49MB/Downloading blended_skill_talk.tar.gz: 19%|1| 7.08M/38.1M [00:06<00:21, 1.46MB/Downloading blended_skill_talk.tar.gz: 19%|1| 7.27M/38.1M [00:07<00:19, 1.56MB/Downloading blended_skill_talk.tar.gz: 20%|1| 7.44M/38.1M [00:07<00:19, 1.57MB/Downloading blended_skill_talk.tar.gz: 20%|2| 7.63M/38.1M [00:07<00:20, 1.49MB/Downloading blended_skill_talk.tar.gz: 21%|2| 7.83M/38.1M [00:07<00:19, 1.54MB/Downloading blended_skill_talk.tar.gz: 21%|2| 8.00M/38.1M [00:07<00:20, 1.50MB/Downloading blended_skill_talk.tar.gz: 22%|2| 8.19M/38.1M [00:07<00:18, 1.60MB/Downloading blended_skill_talk.tar.gz: 22%|2| 8.39M/38.1M [00:07<00:17, 1.68MB/Downloading blended_skill_talk.tar.gz: 23%|2| 8.59M/38.1M [00:07<00:17, 1.67MB/Downloading blended_skill_talk.tar.gz: 23%|2| 8.78M/38.1M [00:07<00:17, 1.72MB/Downloading blended_skill_talk.tar.gz: 24%|2| 8.98M/38.1M [00:08<00:16, 1.77MB/Downloading blended_skill_talk.tar.gz: 24%|2| 9.18M/38.1M [00:08<00:16, 1.75MB/Downloading blended_skill_talk.tar.gz: 25%|2| 9.37M/38.1M [00:08<00:16, 1.70MB/Downloading blended_skill_talk.tar.gz: 25%|2| 9.57M/38.1M [00:08<00:17, 1.66MB/Downloading blended_skill_talk.tar.gz: 26%|2| 9.80M/38.1M [00:08<00:16, 1.71MB/Downloading blended_skill_talk.tar.gz: 26%|2| 9.99M/38.1M [00:08<00:17, 1.57MB/Downloading blended_skill_talk.tar.gz: 27%|2| 10.2M/38.1M [00:08<00:18, 1.47MB/Downloading blended_skill_talk.tar.gz: 27%|2| 10.4M/38.1M [00:08<00:17, 1.62MB/Downloading blended_skill_talk.tar.gz: 28%|2| 10.6M/38.1M [00:09<00:17, 1.58MB/Downloading blended_skill_talk.tar.gz: 28%|2| 10.7M/38.1M [00:09<00:24, 1.12MB/Downloading blended_skill_talk.tar.gz: 29%|2| 11.1M/38.1M [00:09<00:19, 1.39MB/Downloading blended_skill_talk.tar.gz: 30%|2| 11.3M/38.1M [00:09<00:18, 1.44MB/Downloading blended_skill_talk.tar.gz: 30%|3| 11.5M/38.1M [00:09<00:18, 1.42MB/Downloading blended_skill_talk.tar.gz: 31%|3| 11.7M/38.1M [00:09<00:17, 1.55MB/Downloading blended_skill_talk.tar.gz: 31%|3| 11.9M/38.1M [00:09<00:15, 1.65MB/Downloading blended_skill_talk.tar.gz: 32%|3| 12.1M/38.1M [00:10<00:15, 1.72MB/Downloading blended_skill_talk.tar.gz: 32%|3| 12.3M/38.1M [00:10<00:15, 1.67MB/Downloading blended_skill_talk.tar.gz: 33%|3| 12.5M/38.1M [00:10<00:17, 1.43MB/Downloading blended_skill_talk.tar.gz: 33%|3| 12.6M/38.1M [00:10<00:28, 899kB/sDownloading blended_skill_talk.tar.gz: 34%|3| 12.8M/38.1M [00:10<00:23, 1.06MB/Downloading blended_skill_talk.tar.gz: 34%|3| 13.0M/38.1M [00:10<00:21, 1.17MB/Downloading blended_skill_talk.tar.gz: 35%|3| 13.2M/38.1M [00:11<00:19, 1.27MB/Downloading blended_skill_talk.tar.gz: 35%|3| 13.4M/38.1M [00:11<00:17, 1.42MB/Downloading blended_skill_talk.tar.gz: 36%|3| 13.6M/38.1M [00:11<00:16, 1.51MB/Downloading blended_skill_talk.tar.gz: 36%|3| 13.8M/38.1M [00:11<00:14, 1.63MB/Downloading blended_skill_talk.tar.gz: 37%|3| 14.0M/38.1M [00:11<00:15, 1.58MB/Downloading blended_skill_talk.tar.gz: 37%|3| 14.2M/38.1M [00:11<00:14, 1.69MB/Downloading blended_skill_talk.tar.gz: 38%|3| 14.4M/38.1M [00:11<00:14, 1.64MB/Downloading blended_skill_talk.tar.gz: 38%|3| 14.6M/38.1M [00:11<00:13, 1.70MB/Downloading blended_skill_talk.tar.gz: 39%|3| 14.8M/38.1M [00:11<00:14, 1.57MB/Downloading blended_skill_talk.tar.gz: 39%|3| 15.0M/38.1M [00:12<00:15, 1.53MB/Downloading blended_skill_talk.tar.gz: 40%|3| 15.2M/38.1M [00:12<00:14, 1.61MB/Downloading blended_skill_talk.tar.gz: 40%|4| 15.4M/38.1M [00:12<00:13, 1.66MB/Downloading blended_skill_talk.tar.gz: 41%|4| 15.6M/38.1M [00:12<00:13, 1.72MB/Downloading blended_skill_talk.tar.gz: 41%|4| 15.8M/38.1M [00:12<00:14, 1.53MB/Downloading blended_skill_talk.tar.gz: 42%|4| 16.0M/38.1M [00:12<00:13, 1.64MB/Downloading blended_skill_talk.tar.gz: 42%|4| 16.2M/38.1M [00:12<00:13, 1.58MB/Downloading blended_skill_talk.tar.gz: 43%|4| 16.4M/38.1M [00:12<00:12, 1.68MB/Downloading blended_skill_talk.tar.gz: 44%|4| 16.6M/38.1M [00:13<00:12, 1.76MB/Downloading blended_skill_talk.tar.gz: 44%|4| 16.8M/38.1M [00:13<00:11, 1.81MB/Downloading blended_skill_talk.tar.gz: 45%|4| 17.0M/38.1M [00:13<00:12, 1.73MB/Downloading blended_skill_talk.tar.gz: 45%|4| 17.2M/38.1M [00:13<00:11, 1.86MB/Downloading blended_skill_talk.tar.gz: 46%|4| 17.4M/38.1M [00:13<00:12, 1.72MB/Downloading blended_skill_talk.tar.gz: 46%|4| 17.6M/38.1M [00:13<00:11, 1.72MB/Downloading blended_skill_talk.tar.gz: 47%|4| 17.8M/38.1M [00:13<00:11, 1.79MB/Downloading blended_skill_talk.tar.gz: 47%|4| 18.1M/38.1M [00:13<00:10, 1.94MB/Downloading blended_skill_talk.tar.gz: 48%|4| 18.3M/38.1M [00:13<00:09, 2.00MB/Downloading blended_skill_talk.tar.gz: 49%|4| 18.5M/38.1M [00:14<00:09, 1.99MB/Downloading blended_skill_talk.tar.gz: 49%|4| 18.8M/38.1M [00:14<00:09, 2.01MB/Downloading blended_skill_talk.tar.gz: 50%|4| 19.0M/38.1M [00:14<00:09, 1.94MB/Downloading blended_skill_talk.tar.gz: 50%|5| 19.2M/38.1M [00:14<00:09, 2.01MB/Downloading blended_skill_talk.tar.gz: 51%|5| 19.5M/38.1M [00:14<00:09, 1.88MB/Downloading blended_skill_talk.tar.gz: 52%|5| 19.7M/38.1M [00:14<00:09, 1.88MB/Downloading blended_skill_talk.tar.gz: 52%|5| 19.9M/38.1M [00:14<00:09, 1.93MB/Downloading blended_skill_talk.tar.gz: 53%|5| 20.1M/38.1M [00:14<00:09, 1.81MB/Downloading blended_skill_talk.tar.gz: 53%|5| 20.3M/38.1M [00:14<00:09, 1.80MB/Downloading blended_skill_talk.tar.gz: 54%|5| 20.5M/38.1M [00:15<00:09, 1.88MB/Downloading blended_skill_talk.tar.gz: 54%|5| 20.7M/38.1M [00:15<00:09, 1.86MB/Downloading blended_skill_talk.tar.gz: 55%|5| 20.9M/38.1M [00:15<00:08, 1.94MB/Downloading blended_skill_talk.tar.gz: 56%|5| 21.2M/38.1M [00:15<00:08, 1.93MB/Downloading blended_skill_talk.tar.gz: 56%|5| 21.4M/38.1M [00:15<00:09, 1.78MB/Downloading blended_skill_talk.tar.gz: 57%|5| 21.6M/38.1M [00:15<00:08, 1.87MB/Downloading blended_skill_talk.tar.gz: 57%|5| 21.8M/38.1M [00:15<00:08, 1.88MB/Downloading blended_skill_talk.tar.gz: 58%|5| 22.0M/38.1M [00:15<00:09, 1.73MB/Downloading blended_skill_talk.tar.gz: 58%|5| 22.2M/38.1M [00:16<00:10, 1.57MB/Downloading blended_skill_talk.tar.gz: 59%|5| 22.5M/38.1M [00:16<00:09, 1.71MB/Downloading blended_skill_talk.tar.gz: 60%|5| 22.7M/38.1M [00:16<00:08, 1.71MB/Downloading blended_skill_talk.tar.gz: 60%|6| 22.9M/38.1M [00:16<00:09, 1.61MB/Downloading blended_skill_talk.tar.gz: 61%|6| 23.1M/38.1M [00:16<00:09, 1.65MB/Downloading blended_skill_talk.tar.gz: 61%|6| 23.3M/38.1M [00:16<00:08, 1.68MB/Downloading blended_skill_talk.tar.gz: 62%|6| 23.5M/38.1M [00:16<00:09, 1.60MB/Downloading blended_skill_talk.tar.gz: 62%|6| 23.7M/38.1M [00:16<00:08, 1.64MB/Downloading blended_skill_talk.tar.gz: 63%|6| 24.0M/38.1M [00:17<00:07, 1.82MB/Downloading blended_skill_talk.tar.gz: 63%|6| 24.2M/38.1M [00:17<00:09, 1.53MB/Downloading blended_skill_talk.tar.gz: 64%|6| 24.3M/38.1M [00:17<00:08, 1.62MB/Downloading blended_skill_talk.tar.gz: 65%|6| 24.6M/38.1M [00:17<00:07, 1.74MB/Downloading blended_skill_talk.tar.gz: 65%|6| 24.8M/38.1M [00:17<00:07, 1.75MB/Downloading blended_skill_talk.tar.gz: 66%|6| 25.0M/38.1M [00:17<00:08, 1.57MB/Downloading blended_skill_talk.tar.gz: 66%|6| 25.2M/38.1M [00:17<00:07, 1.66MB/Downloading blended_skill_talk.tar.gz: 67%|6| 25.4M/38.1M [00:17<00:07, 1.77MB/Downloading blended_skill_talk.tar.gz: 67%|6| 25.6M/38.1M [00:18<00:07, 1.73MB/Downloading blended_skill_talk.tar.gz: 68%|6| 25.8M/38.1M [00:18<00:06, 1.86MB/Downloading blended_skill_talk.tar.gz: 68%|6| 26.1M/38.1M [00:18<00:06, 1.92MB/Downloading blended_skill_talk.tar.gz: 69%|6| 26.3M/38.1M [00:18<00:06, 1.83MB/Downloading blended_skill_talk.tar.gz: 69%|6| 26.5M/38.1M [00:18<00:06, 1.84MB/Downloading blended_skill_talk.tar.gz: 70%|7| 26.7M/38.1M [00:18<00:06, 1.88MB/Downloading blended_skill_talk.tar.gz: 71%|7| 26.9M/38.1M [00:18<00:06, 1.69MB/Downloading blended_skill_talk.tar.gz: 71%|7| 27.1M/38.1M [00:18<00:06, 1.81MB/Downloading blended_skill_talk.tar.gz: 72%|7| 27.4M/38.1M [00:18<00:05, 1.91MB/Downloading blended_skill_talk.tar.gz: 72%|7| 27.6M/38.1M [00:19<00:05, 1.86MB/Downloading blended_skill_talk.tar.gz: 73%|7| 27.8M/38.1M [00:19<00:05, 1.95MB/Downloading blended_skill_talk.tar.gz: 74%|7| 28.0M/38.1M [00:19<00:05, 1.80MB/Downloading blended_skill_talk.tar.gz: 74%|7| 28.2M/38.1M [00:19<00:05, 1.79MB/Downloading blended_skill_talk.tar.gz: 75%|7| 28.4M/38.1M [00:19<00:05, 1.84MB/Downloading blended_skill_talk.tar.gz: 75%|7| 28.6M/38.1M [00:19<00:05, 1.80MB/Downloading blended_skill_talk.tar.gz: 76%|7| 28.8M/38.1M [00:19<00:05, 1.84MB/Downloading blended_skill_talk.tar.gz: 76%|7| 29.1M/38.1M [00:19<00:04, 1.89MB/Downloading blended_skill_talk.tar.gz: 77%|7| 29.3M/38.1M [00:19<00:04, 1.90MB/Downloading blended_skill_talk.tar.gz: 77%|7| 29.5M/38.1M [00:20<00:04, 1.97MB/Downloading blended_skill_talk.tar.gz: 78%|7| 29.7M/38.1M [00:20<00:04, 1.94MB/Downloading blended_skill_talk.tar.gz: 79%|7| 29.9M/38.1M [00:20<00:04, 2.01MB/Downloading blended_skill_talk.tar.gz: 79%|7| 30.2M/38.1M [00:20<00:04, 1.97MB/Downloading blended_skill_talk.tar.gz: 80%|7| 30.4M/38.1M [00:20<00:04, 1.87MB/Downloading blended_skill_talk.tar.gz: 80%|8| 30.6M/38.1M [00:20<00:06, 1.08MB/Downloading blended_skill_talk.tar.gz: 81%|8| 30.8M/38.1M [00:21<00:06, 1.19MB/Downloading blended_skill_talk.tar.gz: 81%|8| 31.0M/38.1M [00:21<00:05, 1.33MB/Downloading blended_skill_talk.tar.gz: 82%|8| 31.2M/38.1M [00:21<00:04, 1.45MB/Downloading blended_skill_talk.tar.gz: 82%|8| 31.4M/38.1M [00:21<00:04, 1.45MB/Downloading blended_skill_talk.tar.gz: 83%|8| 31.6M/38.1M [00:21<00:04, 1.35MB/Downloading blended_skill_talk.tar.gz: 84%|8| 31.8M/38.1M [00:21<00:04, 1.51MB/Downloading blended_skill_talk.tar.gz: 84%|8| 32.0M/38.1M [00:21<00:03, 1.68MB/Downloading blended_skill_talk.tar.gz: 85%|8| 32.2M/38.1M [00:21<00:03, 1.68MB/Downloading blended_skill_talk.tar.gz: 85%|8| 32.5M/38.1M [00:21<00:02, 1.88MB/Downloading blended_skill_talk.tar.gz: 86%|8| 32.7M/38.1M [00:22<00:02, 1.92MB/Downloading blended_skill_talk.tar.gz: 87%|8| 33.0M/38.1M [00:22<00:02, 1.98MB/Downloading blended_skill_talk.tar.gz: 87%|8| 33.2M/38.1M [00:22<00:03, 1.56MB/Downloading blended_skill_talk.tar.gz: 88%|8| 33.6M/38.1M [00:22<00:02, 1.86MB/Downloading blended_skill_talk.tar.gz: 89%|8| 33.8M/38.1M [00:22<00:02, 1.93MB/Downloading blended_skill_talk.tar.gz: 89%|8| 34.0M/38.1M [00:22<00:01, 2.03MB/Downloading blended_skill_talk.tar.gz: 90%|8| 34.3M/38.1M [00:22<00:02, 1.88MB/Downloading blended_skill_talk.tar.gz: 91%|9| 34.5M/38.1M [00:23<00:01, 1.92MB/Downloading blended_skill_talk.tar.gz: 91%|9| 34.7M/38.1M [00:23<00:01, 1.96MB/Downloading blended_skill_talk.tar.gz: 92%|9| 35.0M/38.1M [00:23<00:01, 2.01MB/Downloading blended_skill_talk.tar.gz: 92%|9| 35.2M/38.1M [00:23<00:01, 2.05MB/Downloading blended_skill_talk.tar.gz: 93%|9| 35.4M/38.1M [00:23<00:01, 2.09MB/Downloading blended_skill_talk.tar.gz: 94%|9| 35.7M/38.1M [00:23<00:01, 2.17MB/Downloading blended_skill_talk.tar.gz: 94%|9| 35.9M/38.1M [00:23<00:01, 2.09MB/Downloading blended_skill_talk.tar.gz: 95%|9| 36.1M/38.1M [00:23<00:01, 1.93MB/Downloading blended_skill_talk.tar.gz: 95%|9| 36.4M/38.1M [00:23<00:00, 2.00MB/Downloading blended_skill_talk.tar.gz: 96%|9| 36.6M/38.1M [00:24<00:00, 2.13MB/Downloading blended_skill_talk.tar.gz: 97%|9| 36.9M/38.1M [00:24<00:00, 2.18MB/Downloading blended_skill_talk.tar.gz: 97%|9| 37.1M/38.1M [00:24<00:00, 1.96MB/Downloading blended_skill_talk.tar.gz: 98%|9| 37.4M/38.1M [00:24<00:00, 2.11MB/Downloading blended_skill_talk.tar.gz: 99%|9| 37.6M/38.1M [00:24<00:00, 2.03MB/Downloading blended_skill_talk.tar.gz: 99%|9| 37.8M/38.1M [00:24<00:00, 1.93MB/Downloading blended_skill_talk.tar.gz: 100%|9| 38.1M/38.1M [00:24<00:00, 1.88MB/Downloading blended_skill_talk.tar.gz: 100%|#| 38.1M/38.1M [00:24<00:00, 1.54MB/s] [ Checksum Successful ] unpacking blended_skill_talk.tar.gz [ downloading: http://parl.ai/downloads/blended_skill_talk/personas_list.txt to /home/weichen/ParlAI/data/blended_skill_talk/persona_list.txt ] Downloading persona_list.txt: 0.00B [00:00, ?B/s] [ Checksum Successful ] [ downloading: http://parl.ai/downloads/blended_skill_talk/topic_to_persona_list.txt to /home/weichen/ParlAI/data/blended_skill_talk/topic_to_persona_list.txt ] Downloading topic_to_persona_list.txt: 0.00B [00:00, ?B/s] [ Checksum Successful ] [ downloading: http://parl.ai/downloads/blended_skill_talk/ed_persona_topicifier__train__both_sides.json to /home/weichen/ParlAI/data/blended_skill_talk/ed_persona_topicifiertrainboth_sides.json ] Downloading ed_persona_topicifiertrainboth_sides.json: 0.00B [00:00, ?B/s] [ Checksum Successful ] [ downloading: http://parl.ai/downloads/blended_skill_talk/ed_persona_topicifier__train__experiencer_only.json to /home/weichen/ParlAI/data/blended_skill_talk/ed_persona_topicifiertrainexperiencer_only.json ] Downloading ed_persona_topicifiertrainexperiencer_only.json: 0.00B [00:00, ?Downloading ed_persona_topicifiertrainexperiencer_only.json: 0.00B [00:00, ?B/s] [ Checksum Successful ] [ downloading: http://parl.ai/downloads/blended_skill_talk/ed_persona_topicifier__valid__experiencer_only.json to /home/weichen/ParlAI/data/blended_skill_talk/ed_persona_topicifiervalidexperiencer_only.json ] Downloading ed_persona_topicifiervalidexperiencer_only.json: 0.00B [00:00, ?Downloading ed_persona_topicifiervalidexperiencer_only.json: 0.00B [00:00, ?B/s] [ Checksum Successful ] [ downloading: http://parl.ai/downloads/blended_skill_talk/ed_persona_topicifier__test__experiencer_only.json to /home/weichen/ParlAI/data/blended_skill_talk/ed_persona_topicifiertestexperiencer_only.json ] Downloading ed_persona_topicifiertestexperiencer_only.json: 0.00B [00:00, ?BDownloading ed_persona_topicifiertestexperiencer_only.json: 0.00B [00:00, ?B/s] [ Checksum Successful ] Loading /home/weichen/ParlAI/data/blended_skill_talk/train.json. Saving to /home/weichen/ParlAI/data/blended_skill_talk/train.txt Loading /home/weichen/ParlAI/data/blended_skill_talk/valid.json. Saving to /home/weichen/ParlAI/data/blended_skill_talk/valid.txt Loading /home/weichen/ParlAI/data/blended_skill_talk/test.json. Saving to /home/weichen/ParlAI/data/blended_skill_talk/test.txt [loading parlAI text data:/home/weichen/ParlAI/data/blended_skill_talk/train.txt] Traceback (most recent call last): File "examples/train_model.py", line 15, in TrainModel.main() File "/home/weichen/ParlAI/parlai/scripts/script.py", line 79, in main return cls._run_args(None) File "/home/weichen/ParlAI/parlai/scripts/script.py", line 66, in _run_args script.run() File "/home/weichen/ParlAI/parlai/scripts/train_model.py", line 759, in run return TrainLoop(self.opt).train() File "/home/weichen/ParlAI/parlai/scripts/train_model.py", line 277, in init self.world = create_task(opt, self.agent) File "/home/weichen/ParlAI/parlai/core/worlds.py", line 1626, in create_task world = MultiWorld(opt, user_agents, default_world=default_world) File "/home/weichen/ParlAI/parlai/core/worlds.py", line 602, in init opt_singletask, agents, default_world=default_world File "/home/weichen/ParlAI/parlai/core/worlds.py", line 1585, in create_task_world task_agents = _create_task_agents(opt) File "/home/weichen/ParlAI/parlai/core/worlds.py", line 1570, in _create_task_agents task_agents = my_module.create_agents(opt) # type: ignore File "/home/weichen/ParlAI/parlai/tasks/blended_skill_talk/agents.py", line 82, in create_agents return create_task_agent_from_taskname(opt) File "/home/weichen/ParlAI/parlai/core/teachers.py", line 2230, in create_task_agent_from_taskname task_agents = teacher_class(opt) File "/home/weichen/ParlAI/parlai/tasks/blended_skill_talk/agents.py", line 63, in init super().init(opt, shared) File "/home/weichen/ParlAI/parlai/core/teachers.py", line 1344, in init self._setup_data(opt.get('parlaidialogteacher_datafile')) File "/home/weichen/ParlAI/parlai/core/teachers.py", line 1396, in _setup_data f'ParlaiDialogTeacher requires a "text" field in every ' ValueError: ParlaiDialogTeacher requires a "text" field in every entry, but one is missing in Line 1 in /home/weichen/ParlAI/data/blended_skill_talk/train.txt. The line is: b"text:your persona: i've 2 kids.\nyour persona: i love flowers.\nI love live music, that's why I try to go to concerts\nI do too. Wat do you like?\nI like acting, I hope to be an actor, what about you?\tlabels:that is ok. have any kids?\tcontext_dataset:empathetic_dialogues\tfree_message:I like acting, I hope to be an actor, what about you?\tconvai2:i love acting ! i'll be famous someday . what do you do ?\tempathetic_dialogues:Any favorite actors?\twizard_of_wikipedia:I would like to develop my acting skills. What are some tips you have to not get nervous?\tguided_chosen_suggestion:"

(parlai_cb) weichen@polarbear:~/ParlAI$ python examples/train_model.py -t blended_skill_talk,wizard_of_wikipedia,convai2:normalized -m transformer/generator --multitask-weights 1,3,3,3 --init-model zoo:tutorial_transformer_generator/model --dict-file zoo:tutorial_transformer_generator/model.dict --embedding-size 512 --n-layers 8 --ffn-size 2048 --dropout 0.1 --n-heads 16 --learn-positional-embeddings True --n-positions 512 --variant xlm --activation gelu --skip-generation True --fp16 True --text-truncate 512 --label-truncate 128 --dict-tokenizer bpe --dict-lower True -lr 1e-06 --optimizer adamax --lr-scheduler reduceonplateau --gradient-clip 0.1 -veps 0.25 --betas 0.9,0.999 --update-freq 1 --attention-dropout 0.0 --relu-dropout 0.0 --skip-generation True -vp 15 -stim 60 -vme 20000 -bs 16 -vmt ppl -vmm min --save-after-valid True --model-file /tmp/test_train_90M [ building dictionary first... ] [ dictionary already built .] [ no model with opt yet at: /tmp/test_train_90M(.opt) ]

[ WARNING ] : your model is being loaded with opts that do not exist in the model you are initializing the weights with: dynamic_batching: None,datapath: /home/weichen/ParlAI/data,label_type: response,include_knowledge: True,include_checked_sentence: True,include_knowledge_separator: False,num_topics: 5,n_encoder_layers: -1,n_decoder_layers: -1,model_parallel: False,beam_length_penalty: 0.65,topk: 10,topp: 0.9,beam_delay: 30,beam_blacklist_filename: None,temperature: 1.0,compute_tokenized_bleu: False,interactive_mode: False,fp16_impl: apex,force_fp16_tokens: False,adafactor_eps: (1e-30, 0.001),history_add_global_end_token: None,bpe_vocab: None,bpe_merge: None,bpe_add_prefix_space: None,max_lr_steps: -1,invsqrt_lr_decay_gamma: -1,parlai_home: /home/weichen/ParlAI

[ WARNING ] : your model is being loaded with opts that differ from the model you are initializing the weights with. Add the following args to your run command to change this:

--task internal:new-reddit:presorted --download-path /private/home/roller/working/parlai/downloads --datatype train:stream --multitask-weights 1 --batchsize 48 --num-epochs 5.0 --validation-every-n-secs 1800.0 --save-every-n-secs -1 --validation-every-n-epochs -1 --validation-max-exs 9920 --short-final-eval True --validation-patience 0 --dict-build-first True --load-from-checkpoint True --numworkers 4 --pytorch-preprocess False --pytorch-teacher-batch-sort False --batch-sort-cache-type pop --batch-length-range 5 --shuffle False --batch-sort-field text --pytorch-context-length -1 --pytorch-include-labels True --log-every-n-secs 30.0 --distributed-world-size 64 --verbose False --port 61337 --beam-size 8 --beam-min-n-best 3 --beam-min-length 10 --skip-generation False --inference beam --optimizer fused-adam --learningrate 0.0005 --gradient-clip 10.0 --adam-eps 1e-06 --betas 0.9,0.98 --weight-decay 0.01 --lr-scheduler invsqrt --warmup-updates 20000 --gpu 0 --beam-block-ngram 3 --beam-context-block-ngram 3

[ Using CUDA ] /home/weichen/ParlAI/parlai/utils/fp16.py:144: UserWarning: You set --fp16 true with --fp16-impl apex, but fp16 with apex is unavailable. To use apex fp16, please install APEX from https://github.com/NVIDIA/apex. 'You set --fp16 true with --fp16-impl apex, but fp16 ' Dictionary: loading dictionary from /home/weichen/ParlAI/data/models/tutorial_transformer_generator/model.dict [ num words = 54944 ] Total parameters: 87,508,992 (87,508,992 trainable) [ Loading existing model params from /home/weichen/ParlAI/data/models/tutorial_transformer_generator/model ] WARNING: not loading optim state since optim class changed. [creating task(s): blended_skill_talk,wizard_of_wikipedia,convai2:normalized] [loading parlAI text data:/home/weichen/ParlAI/data/blended_skill_talk/train.txt] Traceback (most recent call last): File "examples/train_model.py", line 15, in TrainModel.main() File "/home/weichen/ParlAI/parlai/scripts/script.py", line 79, in main return cls._run_args(None) File "/home/weichen/ParlAI/parlai/scripts/script.py", line 66, in _run_args script.run() File "/home/weichen/ParlAI/parlai/scripts/train_model.py", line 759, in run return TrainLoop(self.opt).train() File "/home/weichen/ParlAI/parlai/scripts/train_model.py", line 277, in init self.world = create_task(opt, self.agent) File "/home/weichen/ParlAI/parlai/core/worlds.py", line 1626, in create_task world = MultiWorld(opt, user_agents, default_world=default_world) File "/home/weichen/ParlAI/parlai/core/worlds.py", line 602, in init opt_singletask, agents, default_world=default_world File "/home/weichen/ParlAI/parlai/core/worlds.py", line 1585, in create_task_world task_agents = _create_task_agents(opt) File "/home/weichen/ParlAI/parlai/core/worlds.py", line 1570, in _create_task_agents task_agents = my_module.create_agents(opt) # type: ignore File "/home/weichen/ParlAI/parlai/tasks/blended_skill_talk/agents.py", line 82, in create_agents return create_task_agent_from_taskname(opt) File "/home/weichen/ParlAI/parlai/core/teachers.py", line 2230, in create_task_agent_from_taskname task_agents = teacher_class(opt) File "/home/weichen/ParlAI/parlai/tasks/blended_skill_talk/agents.py", line 63, in init super().init(opt, shared) File "/home/weichen/ParlAI/parlai/core/teachers.py", line 1344, in init self._setup_data(opt.get('parlaidialogteacher_datafile')) File "/home/weichen/ParlAI/parlai/core/teachers.py", line 1396, in _setup_data f'ParlaiDialogTeacher requires a "text" field in every ' ValueError: ParlaiDialogTeacher requires a "text" field in every entry, but one is missing in Line 1 in /home/weichen/ParlAI/data/blended_skill_talk/train.txt. The line is: b"text:your persona: i've 2 kids.\nyour persona: i love flowers.\nI love live music, that's why I try to go to concerts\nI do too. Wat do you like?\nI like acting, I hope to be an actor, what about you?\tlabels:that is ok. have any kids?\tcontext_dataset:empathetic_dialogues\tfree_message:I like acting, I hope to be an actor, what about you?\tconvai2:i love acting ! i'll be famous someday . what do you do ?\tempathetic_dialogues:Any favorite actors?\twizard_of_wikipedia:I would like to develop my acting skills. What are some tips you have to not get nervous?\tguided_chosen_suggestion:"

Additional context Add any other context about the problem here. (like proxy settings, network setup, overall goals, etc.)

stephenroller commented 4 years ago

Thanks for reporting

stephenroller commented 4 years ago

Did you modify anything? I just pulled from master and don't see this happening.

Dogy06 commented 4 years ago

I changed line 112 in parlai/tasks/blended_skill_talk/build.py to resolve an encoding issue but other than that I cloned from master: The code changed from: f_write.write(f'{line} \n') to: f_write.write(f'{line.encode("utf-8")} \n')

The file writer couldnt write the string in latin-1 and outputted an encoding error so I converted it into utf-8: UnicodeEncodeError: 'latin-1' codec can't encode character '\u2019' in position 271: ordinal not in range(256) Normally, I would just encode these lines in a different encoding (utf-8). I am running Linux Ubuntu 18.04. I think this is the problem. Any suggestions on fixing this would be greatly appreciated. Thanks.

Dogy06 commented 4 years ago

Fixed with export LANG=en_US.UTF-8 in bashrc