xinwei96
commented
2 years ago 您好，仅从日志中的报错我也不能确定是什么原因，我猜测是您没有对词表进行修改。 因为我把“speaker1”.. 作为special token，所以需要把词表中的[used1]..[used8]替换为speaker1...speaker8. 您也可以把‘./data/vocab.txt’ 替换bert原始的vocab.txt
Open topDreamer opened 2 years ago
xinwei96
commented
2 years ago 您好，仅从日志中的报错我也不能确定是什么原因，我猜测是您没有对词表进行修改。 因为我把“speaker1”.. 作为special token，所以需要把词表中的[used1]..[used8]替换为speaker1...speaker8. 您也可以把‘./data/vocab.txt’ 替换bert原始的vocab.txt
xinwei96
commented
2 years ago 修改后，请记得先删除data目录下之前生成的“*.p”文件，再重新运行。
topDreamer
commented
2 years ago 非常感谢@xinwei 的认真答疑和帮助！我上面提供的日志，跑之前是用‘./data/vocab.txt’ 替换bert原始的vocab.txt的，现在我建了一个虚拟环境，所有的环境参数都和你给出的是一样的，现在的报错应该还是分布式的错误，我还在找错误的解决方案，我考虑把分布式换成单卡训练，看看是否有效～ 错误日志如下：
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1607370144807/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f704b3db8b2 in /home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7f704b62d982 in /home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f704b3c6b7d in /home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3:
Traceback (most recent call last):
File "/home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/zhangs-b20/data/miniconda3/envs/MRC/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
xinwei96
commented
2 years ago 啊这不是吧，当我把词表换成bert原始的时，也出现了一样的错误。 麻烦您确认一下，在替换词表后，把之前生成的train.p, dev.p, test.p都删掉再重新运行。 我建了一个pytorch-1.7.1的虚拟环境，代码是可以运行的，但可能随机数生成的机制有差异，结果与之前的略有不同。
xinwei96
commented
2 years ago 或者您用单机跑，debug模式，看下tokenizer输出的“speaker1”对应的index是否是1，如果不是，就说明是词表的问题
topDreamer
commented
2 years ago 感谢@xinwei，经测试是apex的版本问题，已经解决跑通了。 按照paper参数跑实验，目前最好一轮的测试集效果目前是71.9左右（可能随机数生成的机制有差异，结果与之前的略有不同）然后我测试一下去掉Relational Reasoning Regularization模块（具体操作：令sigma=0），看log日志中最好一轮的测试集上F1=72.6。 麻烦@xinwei，另外想请教一下： 1、我去掉R3的操作（具体操作：直接令sigma=0，原文是0.1）是否存在问题呢？ 2、另外实验过程中指标比较震荡，我是挑选的最好一轮的测试集的F1=71.9，这样是否存在问题？（PS：模型最终打印的是验证集上最好的一轮效果，我上面的指标给的是测试集上最好的一轮结果）
xinwei96
commented
2 years ago paper中report的是在验证集上最优的模型 在验证集和测试集上的结果。 1、去掉R3（直接令sigma=0）没有问题。 2、实验过程中指标震荡的原因有两个：（1）dialogRE数据集太小了，类别的数量也极度不均衡。（2）学习率的变化采用了CosineAnnealingLR（学习率呈周期变化），从而导致了实验过程中的指标震荡。（另，baseline中的GAIN也采用了这种变化方式。
这个数据集本身太小了，确实最后的结果会有差异。在torch-1.7.1的环境下我的结果是 eval F1=71.04，test F1=72.03 在torch-1.2下我的结果是 eval F1=71.63，test F1=72.19
感谢开源非常好的工作！ 我使用pytorch==1.7.1，两块V100跑这份代码，参数完全按照papers里面的设置，跑后的错误提示应该是分布式的错误，尝试解决没有解决该问题，特来请教！log日志如下：
Namespace(batch_size=2, bert_output_size=768, bert_path='/home/data/bert-base-chinese', dev_path='../data/dev.json', device_name='cuda', dropout_rate=0.2, epoch_nums=60, eval_batch_size=2, ft_lr=2e-05, handle_abbr=True, local_rank=0, log_path='../logs/', lower=True, lr=0.0005, max_input_lens=512, max_seq_lens=725, max_utter_nums=42, mode='train', num_heads=12, offset=256, optimal_model_path='./model/best.pkl', output_path='../model/', preds_output_path='./results.json', rel_nums=36, report_every_batch=50, rule_nums=24, save_dir='../model/', seed=0, sigma=0.1, t_max=10, test_batch_size=1, test_path='../data/test.json', threshold=0.5, train_path='../data/train.json', type_dict_path='../data/entity_type_id.json', weight_decay=1e-07) Load dialogre dataset... (This can take some time) loading files... Load dialogre dataset... (This can take some time) loading files... all done. all done. Start Training: 1 Traceback (most recent call last): File "main.py", line 291, in
main()
File "main.py", line 287, in main
train(args)
File "main.py", line 95, in train
preds = model(batch)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
output = self.module(*inputs[0], *kwargs[0])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/raid/CoIn_dialogRE-main/src/model.py", line 48, in forward
input_feat, input_mask = self.encoder(input_tokens_1, input_tokens_2, input_mask_1, input_mask_2, batch_size, max_seq_lens)
File "/raid/CoIn_dialogRE-main/src/model.py", line 107, in encoder
input_feat = self.bert_encoder(input_tokens_1, attention_mask=input_mask_1)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
Traceback (most recent call last):
File "main.py", line 291, in
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 710, in forward
main()
File "main.py", line 287, in main
head_mask=head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
train(args)
File "main.py", line 95, in train
preds = model(batch)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, *kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 431, in forward
layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 409, in forward
output = self.module(*inputs[0], *kwargs[0])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/raid/CoIn_dialogRE-main/src/model.py", line 48, in forward
input_feat, input_mask = self.encoder(input_tokens_1, input_tokens_2, input_mask_1, input_mask_2, batch_size, max_seq_lens)
File "/raid/CoIn_dialogRE-main/src/model.py", line 114, in encoder
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 366, in forward
input1 = self.bert_encoder(input_tokens_1, attention_mask=input_mask_1)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
self_outputs = self.self(input_tensor, attention_mask, head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, *kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 710, in forward
result = self.forward(input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 286, in forward
head_mask=head_mask)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
mixed_query_layer = self.query(hidden_states)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 431, in forward
result = self.forward(*input, *kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward
return F.linear(input, self.weight, self.bias)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/functional.py", line 1692, in linear
layer_outputs = layer_module(hidden_states, attention_mask, head_mask[i])
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(input, kwargs)
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 409, in forward
attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
output = input.matmul(weight.t()) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling + 0x5fea0a (0x7fe320aafa0a in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x5feab6 (0x7fe320aafab6 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x1a3f6e (0x5563cb729f6e in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #6: + 0x10e34c (0x5563cb69434c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #7: + 0x10e4a7 (0x5563cb6944a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #8: + 0x10e4a7 (0x5563cb6944a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #9: + 0xfd9c8 (0x5563cb6839c8 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #10: + 0x10eb77 (0x5563cb694b77 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #11: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #12: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #13: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #14: + 0x10eb8d (0x5563cb694b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #15: PyDict_SetItem + 0x502 (0x5563cb6e9da2 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #16: PyDict_SetItemString + 0x4f (0x5563cb6ea86f in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #17: PyImport_Cleanup + 0xa0 (0x5563cb7305d0 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #18: Py_FinalizeEx + 0x67 (0x5563cb7ab487 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #19: + 0x237f03 (0x5563cb7bdf03 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #20: _Py_UnixMain + 0x3c (0x5563cb7be22c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7fe345776b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: + 0x1dce90 (0x5563cb762e90 in /home/data/miniconda3/envs/DeepEnv/bin/python)
cublasCreate(handle)result = self.forward(*input, kwargs) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 366, in forward self_outputs = self.self(input_tensor, attention_mask, head_mask) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(*input, *kwargs) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/pytorch_transformers/modeling_bert.py", line 286, in forward mixed_query_layer = self.query(hidden_states) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl result = self.forward(input, kwargs) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 93, in forward return F.linear(input, self.weight, self.bias) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/nn/functional.py", line 1692, in linear output = input.matmul(weight.t()) RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when callingcublasCreate(handle)terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1607370144807/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fe2eaaa58b2 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fe2eacf7982 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fe2eaa90b7d in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #3:terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered Exception raised from create_event_internal at /opt/conda/conda-bld/pytorch_1607370144807/work/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7efe812068b2 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7efe81458982 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10_cuda.so) frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7efe811f1b7d in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libc10.so) frame #3: + 0x5fea0a (0x7efeb7210a0a in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x5feab6 (0x7efeb7210ab6 in /home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x1a3f6e (0x55dfc4a9cf6e in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #6: + 0x10e34c (0x55dfc4a0734c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #7: + 0x10e4a7 (0x55dfc4a074a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #8: + 0x10e4a7 (0x55dfc4a074a7 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #9: + 0xfd9c8 (0x55dfc49f69c8 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #10: + 0x10eb77 (0x55dfc4a07b77 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #11: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #12: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #13: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #14: + 0x10eb8d (0x55dfc4a07b8d in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #15: PyDict_SetItem + 0x502 (0x55dfc4a5cda2 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #16: PyDict_SetItemString + 0x4f (0x55dfc4a5d86f in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #17: PyImport_Cleanup + 0xa0 (0x55dfc4aa35d0 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #18: Py_FinalizeEx + 0x67 (0x55dfc4b1e487 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #19: + 0x237f03 (0x55dfc4b30f03 in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #20: _Py_UnixMain + 0x3c (0x55dfc4b3122c in /home/data/miniconda3/envs/DeepEnv/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7efedbed7b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: + 0x1dce90 (0x55dfc4ad5e90 in /home/data/miniconda3/envs/DeepEnv/bin/python)
Traceback (most recent call last): File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in
main()
File "/home/data/miniconda3/envs/DeepEnv/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/home/data/miniconda3/envs/DeepEnv/bin/python', '-u', 'main.py', '--local_rank=1']' died with <Signals.SIGABRT: 6>.
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.