Open topDreamer opened 2 years ago
您好,仅从日志中的报错我也不能确定是什么原因,我猜测是您没有对词表进行修改。 因为我把“speaker1”.. 作为special token,所以需要把词表中的[used1]..[used8]替换为speaker1...speaker8. 您也可以把‘./data/vocab.txt’ 替换bert原始的vocab.txt
修改后,请记得先删除data目录下之前生成的“*.p”文件,再重新运行。
非常感谢@xinwei 的认真答疑和帮助!我上面提供的日志,跑之前是用‘./data/vocab.txt’ 替换bert原始的vocab.txt的,现在我建了一个虚拟环境,所有的环境参数都和你给出的是一样的,现在的报错应该还是分布式的错误,我还在找错误的解决方案,我考虑把分布式换成单卡训练,看看是否有效~ 错误日志如下:
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1607370144807/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f704b3db8b2 in /home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7f704b62d982 in /home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f704b3c6b7d in /home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3:
Traceback (most recent call last):
File "/home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
啊这不是吧,当我把词表换成bert原始的时,也出现了一样的错误。 麻烦您确认一下,在替换词表后,把之前生成的train.p, dev.p, test.p都删掉再重新运行。 我建了一个pytorch-1.7.1的虚拟环境,代码是可以运行的,但可能随机数生成的机制有差异,结果与之前的略有不同。
或者您用单机跑,debug模式,看下tokenizer输出的“speaker1”对应的index是否是1,如果不是,就说明是词表的问题
感谢@xinwei,经测试是apex的版本问题,已经解决跑通了。 按照paper参数跑实验,目前最好一轮的测试集效果目前是71.9左右(可能随机数生成的机制有差异,结果与之前的略有不同)然后我测试一下去掉Relational Reasoning Regularization模块(具体操作:令sigma=0),看log日志中最好一轮的测试集上F1=72.6。 麻烦@xinwei,另外想请教一下: 1、我去掉R3的操作(具体操作:直接令sigma=0,原文是0.1)是否存在问题呢? 2、另外实验过程中指标比较震荡,我是挑选的最好一轮的测试集的F1=71.9,这样是否存在问题?(PS:模型最终打印的是验证集上最好的一轮效果,我上面的指标给的是测试集上最好的一轮结果)
paper中report的是在验证集上最优的模型 在验证集和测试集上的结果。 1、去掉R3(直接令sigma=0)没有问题。 2、实验过程中指标震荡的原因有两个:(1)dialogRE数据集太小了,类别的数量也极度不均衡。(2)学习率的变化采用了CosineAnnealingLR(学习率呈周期变化),从而导致了实验过程中的指标震荡。(另,baseline中的GAIN也采用了这种变化方式。
这个数据集本身太小了,确实最后的结果会有差异。在torch-1.7.1的环境下我的结果是 eval F1=71.04,test F1=72.03 在torch-1.2下我的结果是 eval F1=71.63,test F1=72.19
感谢开源非常好的工作! 我使用pytorch==1.7.1,两块V100跑这份代码,参数完全按照papers里面的设置,跑后的错误提示应该是分布式的错误,尝试解决没有解决该问题,特来请教!log日志如下:
Namespace(batch_size=2, bert_output_size=768, bert_path='/home/data/bert-base-chinese', dev_path='../data/dev.json', device_name='cuda', dropout_rate=0.2, epoch_nums=60, eval_batch_size=2, ft_lr=2e-05, handle_abbr=True, local_rank=0, log_path='../logs/', lower=True, lr=0.0005, max_input_lens=512, max_seq_lens=725, max_utter_nums=42, mode='train', num_heads=12, offset=256, optimal_model_path='./model/best.pkl', output_path='../model/', preds_output_path='./results.json', rel_nums=36, report_every_batch=50, rule_nums=24, save_dir='../model/', seed=0, sigma=0.1, t_max=10, test_batch_size=1, test_path='../data/test.json', threshold=0.5, train_path='../data/train.json', type_dict_path='../data/entity_type_id.json', weight_decay=1e-07) Load dialogre dataset... (This can take some time) loading files... Load dialogre dataset... (This can take some time) loading files... all done. all done. Start Training: 1 Traceback (most recent call last): File "main.py", line 291, in
main()
File "main.py", line 287, in main
train(args)
File "main.py", line 95, in train
preds = model(batch)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
output = self.module(*inputs[0], *kwargs[0])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/raid/CoIn_dialogRE-main/src/model.py", line 48, in forward
input_feat, input_mask = self.encoder(input_tokens_1, input_tokens_2, input_mask_1, input_mask_2, batch_size, max_seq_lens)
File "/raid/CoIn_dialogRE-main/src/model.py", line 107, in encoder
input_feat = self.bert_encoder(input_tokens_1, attention_mask=input_mask_1)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
Traceback (most recent call last):
File "main.py", line 291, in
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 710, in forward
main()
File "main.py", line 287, in main
head_mask=head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
train(args)
File "main.py", line 95, in train
preds = model(batch)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, *kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 431, in forward
layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 409, in forward
output = self.module(*inputs[0], *kwargs[0])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/raid/CoIn_dialogRE-main/src/model.py", line 48, in forward
input_feat, input_mask = self.encoder(input_tokens_1, input_tokens_2, input_mask_1, input_mask_2, batch_size, max_seq_lens)
File "/raid/CoIn_dialogRE-main/src/model.py", line 114, in encoder
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 366, in forward
input1 = self.bert_encoder(input_tokens_1, attention_mask=input_mask_1)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
self_outputs = self.self(input_tensor, attention_mask, head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, *kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 710, in forward
result = self.forward(input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 286, in forward
head_mask=head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
mixed_query_layer = self.query(hidden_states)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 431, in forward
result = self.forward(*input, *kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/functional.py", line 1692, in linear
layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 409, in forward
attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
output = input.matmul(weight.t()) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling + 0x5fea0a (0x7fe320aafa0a in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x5feab6 (0x7fe320aafab6 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x1a3f6e (0x5563cb729f6e in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #6: + 0x10e34c (0x5563cb69434c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #7: + 0x10e4a7 (0x5563cb6944a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #8: + 0x10e4a7 (0x5563cb6944a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #9: + 0xfd9c8 (0x5563cb6839c8 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #10: + 0x10eb77 (0x5563cb694b77 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #11: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #12: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #13: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #14: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #15: PyDict_SetItem + 0x502 (0x5563cb6e9da2 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #16: PyDict_SetItemString + 0x4f (0x5563cb6ea86f in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #17: PyImport_Cleanup + 0xa0 (0x5563cb7305d0 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #18: Py_FinalizeEx + 0x67 (0x5563cb7ab487 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #19: + 0x237f03 (0x5563cb7bdf03 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #20: _Py_UnixMain + 0x3c (0x5563cb7be22c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7fe345776b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: + 0x1dce90 (0x5563cb762e90 in /home/data/miniconda3/envs/DeepEnv/bin/python)
cublasCreate(handle)
result = self.forward(*input, kwargs) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 366, in forward self_outputs = self.self(input_tensor, attention_mask, head_mask) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, *kwargs) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 286, in forward mixed_query_layer = self.query(hidden_states) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, kwargs) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward return F.linear(input, self.weight, self.bias) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/functional.py", line 1692, in linear output = input.matmul(weight.t()) RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when callingcublasCreate(handle)
terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1607370144807/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fe2eaaa58b2 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fe2eacf7982 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fe2eaa90b7d in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #3:terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1607370144807/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7efe812068b2 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7efe81458982 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7efe811f1b7d in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #3: + 0x5fea0a (0x7efeb7210a0a in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x5feab6 (0x7efeb7210ab6 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x1a3f6e (0x55dfc4a9cf6e in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #6: + 0x10e34c (0x55dfc4a0734c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #7: + 0x10e4a7 (0x55dfc4a074a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #8: + 0x10e4a7 (0x55dfc4a074a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #9: + 0xfd9c8 (0x55dfc49f69c8 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #10: + 0x10eb77 (0x55dfc4a07b77 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #11: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #12: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #13: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #14: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #15: PyDict_SetItem + 0x502 (0x55dfc4a5cda2 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #16: PyDict_SetItemString + 0x4f (0x55dfc4a5d86f in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #17: PyImport_Cleanup + 0xa0 (0x55dfc4aa35d0 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #18: Py_FinalizeEx + 0x67 (0x55dfc4b1e487 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #19: + 0x237f03 (0x55dfc4b30f03 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #20: _Py_UnixMain + 0x3c (0x55dfc4b3122c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7efedbed7b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: + 0x1dce90 (0x55dfc4ad5e90 in /home/data/miniconda3/envs/DeepEnv/bin/python)
Traceback (most recent call last): File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
main()
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/home/data/miniconda3/envs/DeepEnv/bin/python', '-u', 'main.py', '--local_rank=1']' died with <Signals.SIGABRT: 6>.
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.