search

LeeSureman / Flat-Lattice-Transformer

code for ACL 2020 paper: FLAT: Chinese NER Using Flat-Lattice Transformer
1k stars 178 forks source link

仿照msar训练自己的数据集出现的问题 #113

Open EleganceM opened 2 years ago

EleganceM commented 2 years ago

C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [27,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [28,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [30,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [31,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. Traceback (most recent call last): File "D:/B/Flat-Lattice-Transformer-master/V0/flat_main.py", line 641, in trainer.train() File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 616, in train raise e File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 609, in train self._train() File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 664, in _train prediction = self._data_forward(self.model, batch_x) File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 752, in _data_forward y = network(x) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forwardcall(*input, **kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\models.py", line 489, in forward print=(self.batch_num==327)) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\modules.py", line 1341, in forward rel_pos_embedding=rel_posembedding,print=print_) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\modules.py", line 1166, in forward rel_pos_embedding=rel_pos_embedding) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(input, **kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\modules.py", line 367, in forward mask = seq_len_to_mask(seq_len+lex_num).bool().unsqueeze(1).unsqueeze(1) File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\utils.py", line 655, in seq_len_to_mask broad_cast_seq_len = torch.arange(max_len).expand(batch_size, -1).to(seq_len) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Process finished with exit code 1

nlper01 commented 1 year ago

C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [27,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [28,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [30,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. C:/w/b/windows/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:93: block: [3414,0,0], thread: [31,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. Traceback (most recent call last): File "D:/B/Flat-Lattice-Transformer-master/V0/flat_main.py", line 641, in trainer.train() File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 616, in train raise e File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 609, in train self._train() File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 664, in _train prediction = self._data_forward(self.model, batch_x) File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\trainer.py", line 752, in _data_forward y = network(x) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forwardcall(*input, **kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\models.py", line 489, in forward print=(self.batch_num==327)) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\modules.py", line 1341, in forward rel_pos_embedding=rel_posembedding,print=print_) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(*input, *kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\modules.py", line 1166, in forward rel_pos_embedding=rel_pos_embedding) File "E:\anaconda\envs\FLAT\lib\site-packages\torch\nn\modules\module.py", line 1102, in _call_impl return forward_call(input, **kwargs) File "D:\B\Flat-Lattice-Transformer-master\V0\modules.py", line 367, in forward mask = seq_len_to_mask(seq_len+lex_num).bool().unsqueeze(1).unsqueeze(1) File "E:\anaconda\envs\FLAT\lib\site-packages\fastNLP\core\utils.py", line 655, in seq_len_to_mask broad_cast_seq_len = torch.arange(max_len).expand(batch_size, -1).to(seq_len) RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Process finished with exit code 1

请问你解决了吗？我也遇到了类似的问题

C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [98287,0,0], thread: [92,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [98287,0,0], thread: [93,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [98287,0,0], thread: [94,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\aten\src\ATen\native\cuda\IndexKernel.cu:91: block: [98287,0,0], thread: [95,0,0] Assertionindex >= -sizes[i] && index < sizes[i] && "index out of bounds"failed. Traceback (most recent call last): File "F:\#CNER\Flat-Lattice-Transformer\V0\flat_main.py", line 637, in <module> trainer.train() File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\fastNLP\core\trainer.py", line 616, in train raise e File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\fastNLP\core\trainer.py", line 609, in train self._train() File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\fastNLP\core\trainer.py", line 664, in _train prediction = self._data_forward(self.model, batch_x) File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\fastNLP\core\trainer.py", line 752, in _data_forward y = network(**x) File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "F:\#CNER\Flat-Lattice-Transformer\V0\models.py", line 488, in forward encoded = self.encoder(embedding,seq_len,lex_num=lex_num,pos_s=pos_s,pos_e=pos_e, File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "F:\#CNER\Flat-Lattice-Transformer\V0\modules.py", line 1333, in forward rel_pos_embedding = self.four_pos_fusion_embedding(pos_s,pos_e) File "D:\Anaconda3\envs\CNER_1.0\lib\site-packages\torch\nn\modules\module.py", line 1110, in _call_impl return forward_call(*input, **kwargs) File "F:\#CNER\Flat-Lattice-Transformer\V0\modules.py", line 93, in forward pe_ss = self.pe_ss[(pos_ss).view(-1)+self.max_seq_len].view(size=[batch,max_seq_len,max_seq_len,-1]) RuntimeError: CUDA error: device-side assert triggered