How to fix this problem CUDA error: device-side assert triggered

[Takeru0618]

Traceback (most recent call last): File "run_train.py", line 428, in main() File "run_train.py", line 417, in main trainer.train(model_path=model_path) File "/home/users/SARG/finetune_trainer.py", line 429, in train tr_loss += self._training_step(model, inputs, optimizer) File "/home/users/SARG/finetune_trainer.py", line 540, in _training_step outputs = model(*inputs) File "/home/users/miniconda3/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in call result = self.forward(input, **kwargs) File "/home/users/SARG/modeling_sarg.py", line 397, in forward outputs = self._forward(inputs) File "/home/users/SARG/modeling_sarg.py", line 296, in _forward tm_mask = self._join_mask(ctx_mask, src_mask) File "/home/users/SARG/modeling_sarg.py", line 256, in _join_mask mask_ru = torch.ones(batch_size, seq_len1, seq_len2, dtype=torch.long).to(mask_lu.device) RuntimeError: CUDA error: device-side assert triggered