leileilin
commented
3 years ago i have solved that. It's really a data problem.
Closed leileilin closed 3 years ago
leileilin
commented
3 years ago i have solved that. It's really a data problem.
Describe the bug when i run the usage of bert like openkiwi doucmentation suggests,
run_info = train_from_file('bert.yaml'). i have encouterd the following problem.Epoch 1: 12%|███████▎ | 482/4006 [01:50<13:31, 4.34it/s, loss=319.510, PEARSON+target_tags_F1_MULT+WMT19_MCC=0.915]Traceback (most recent call last): File "bert_train.py", line 6, in
run_info = train_from_file('bert.yaml')
File "/home/ll/OpenKiwi-master/kiwi/lib/train.py", line 236, in train_from_file
return train_from_configuration(config)
File "/home/ll/OpenKiwi-master/kiwi/lib/train.py", line 249, in train_from_configuration
train_info = run(config)
File "/home/ll/OpenKiwi-master/kiwi/lib/train.py", line 406, in run
trainer.fit(system)
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1003, in fit
results = self.single_gpu_train(model)
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 186, in single_gpu_train
results = self.run_pretrain_routine(model)
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1213, in run_pretrain_routine
self.train()
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 370, in train
self.run_training_epoch()
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 440, in run_training_epoch
enumerate(_with_is_last(train_dataloader)), "get_train_batch"
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/profiler/profilers.py", line 64, in profile_iterable
value = next(iterator)
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 1013, in _with_is_last
for val in it:
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 363, in next
data = self._next_data()
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 989, in _next_data
return self._process_data(data)
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1014, in _process_data
data.reraise()
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/torch/_utils.py", line 395, in reraise
raise self.exc_type(msg)
TypeError: Caught TypeError in DataLoader worker process 3.
Original Traceback (most recent call last):
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 185, in _worker_loop
data = fetcher.fetch(index)
File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/ll/OpenKiwi-master/kiwi/data/encoders/wmt_qe_data_encoder.py", line 329, in collate_fn
for column, examples in columns_data.items()
File "/home/ll/OpenKiwi-master/kiwi/data/encoders/wmt_qe_data_encoder.py", line 329, in
for column, examples in columns_data.items()
File "/home/ll/OpenKiwi-master/kiwi/data/encoders/field_encoders.py", line 241, in batch_encode
batch = [self.encode(example) for example in iterator]
File "/home/ll/OpenKiwi-master/kiwi/data/encoders/field_encoders.py", line 241, in
batch = [self.encode(example) for example in iterator]
File "/home/ll/OpenKiwi-master/kiwi/data/encoders/field_encoders.py", line 232, in encode
matrix += 1
TypeError: can only concatenate tuple (not "int") to tuple
Exception ignored in: <function tqdm.del at 0x7ff55198b680> Traceback (most recent call last): File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/tqdm/std.py", line 1145, in del File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/tqdm/std.py", line 1299, in close File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/tqdm/std.py", line 1492, in display File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/tqdm/std.py", line 1148, in str File "/home/ll/anaconda3/envs/qe/lib/python3.7/site-packages/tqdm/std.py", line 1450, in format_dict TypeError: cannot unpack non-iterable NoneType object
To Reproduce here are my yamls. bert.txt ccmt20.qe.zh_en.txt
Environment (please complete the following information):