2024-05-13 15:45:01,062 - INFO - Save at: ./work_dirs/stpls3d/isbnet_backbone_stpls3d/ws
2024-05-13 15:45:03,147 - INFO - Total params: 7532252
2024-05-13 15:45:03,147 - INFO - Trainable params: 7532252
2024-05-13 15:45:03,151 - INFO - Load train dataset: 2133 scans
2024-05-13 15:45:03,151 - INFO - Load test dataset: 12 scans
2024-05-13 15:45:03,152 - INFO - Scale LR from 0.004 (batch size 16) to 0.002 (batch size 8)
2024-05-13 15:45:03,153 - INFO - Training
../aten/src/ATen/native/cuda/Loss.cu:271: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [29,0,0] Assertion t >= 0 && t < n_classes failed.
Traceback (most recent call last):
File "/home/wang_yongjun/ws/code/ISBNet-master/tools/train.py", line 307, in
main()
File "/home/wang_yongjun/ws/code/ISBNet-master/tools/train.py", line 298, in main
train(epoch, model, optimizer, scheduler, scaler, train_loader, cfg, logger, writer)
File "/home/wang_yongjun/ws/code/ISBNet-master/tools/train.py", line 73, in train
loss, log_vars = model(batch, return_loss=True, epoch=epoch - 1)
File "/home/wang_yongjun/anaconda3/envs/isbnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(input, kwargs)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/isbnet.py", line 217, in forward
return self.forward_train(batch, epoch=epoch)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/util/utils.py", line 172, in wrapper
return func(new_args, *new_kwargs)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/isbnet.py", line 293, in forward_train
losses = self.criterion(batch_inputs, model_outputs)
File "/home/wang_yongjun/anaconda3/envs/isbnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(input, **kwargs)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/criterion.py", line 296, in forward
point_wise_loss = self.cal_point_wise_loss(
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/criterion.py", line 152, in cal_point_wise_loss
semantic_loss = F.cross_entropy(
File "/home/wang_yongjun/anaconda3/envs/isbnet/lib/python3.8/site-packages/torch/nn/functional.py", line 3014, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: device-side assert triggered
Hello, when I take the first step to train my own data, the above problem occurs.How can I solve this problem, please?
2024-05-13 15:45:01,062 - INFO - Save at: ./work_dirs/stpls3d/isbnet_backbone_stpls3d/ws 2024-05-13 15:45:03,147 - INFO - Total params: 7532252 2024-05-13 15:45:03,147 - INFO - Trainable params: 7532252 2024-05-13 15:45:03,151 - INFO - Load train dataset: 2133 scans 2024-05-13 15:45:03,151 - INFO - Load test dataset: 12 scans 2024-05-13 15:45:03,152 - INFO - Scale LR from 0.004 (batch size 16) to 0.002 (batch size 8) 2024-05-13 15:45:03,153 - INFO - Training ../aten/src/ATen/native/cuda/Loss.cu:271: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [29,0,0] Assertion
main()
File "/home/wang_yongjun/ws/code/ISBNet-master/tools/train.py", line 298, in main
train(epoch, model, optimizer, scheduler, scaler, train_loader, cfg, logger, writer)
File "/home/wang_yongjun/ws/code/ISBNet-master/tools/train.py", line 73, in train
loss, log_vars = model(batch, return_loss=True, epoch=epoch - 1)
File "/home/wang_yongjun/anaconda3/envs/isbnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(input, kwargs)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/isbnet.py", line 217, in forward
return self.forward_train(batch, epoch=epoch)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/util/utils.py", line 172, in wrapper
return func(new_args, *new_kwargs)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/isbnet.py", line 293, in forward_train
losses = self.criterion(batch_inputs, model_outputs)
File "/home/wang_yongjun/anaconda3/envs/isbnet/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(input, **kwargs)
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/criterion.py", line 296, in forward
point_wise_loss = self.cal_point_wise_loss(
File "/home/wang_yongjun/zl/Code/ISBNet/isbnet/model/criterion.py", line 152, in cal_point_wise_loss
semantic_loss = F.cross_entropy(
File "/home/wang_yongjun/anaconda3/envs/isbnet/lib/python3.8/site-packages/torch/nn/functional.py", line 3014, in cross_entropy
return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: device-side assert triggered
Hello, when I take the first step to train my own data, the above problem occurs.How can I solve this problem, please?
t >= 0 && t < n_classes
failed. Traceback (most recent call last): File "/home/wang_yongjun/ws/code/ISBNet-master/tools/train.py", line 307, in