CUDA error: invalid configuration argument

Hi, thanks for your great work. I tried using another model to predict 4d occupancy and use dvr.cu for rendering. But after training a few iterations, the following problem occurs, can you provide some suggestions to help solve this bug.

2023-05-30 09:50:43,139 - mmdet - INFO - Epoch [1][6/27]        lr: 1.700e-04, eta: 0:07:41, time: 0.924, data_time: 0.030, memory: 7264, loss_heatmap: 8808.0361, layer_-1_loss_cls: 44.7087, layer_-1_loss_bbox: 11.4984, matched_ious: 0.0000, loss_l1: 15.6480, loss_l2: 294.7178, loss_absrel: 0.8354, loss: 9175.4443, grad_norm: 34641.0859
2023-05-30 09:50:44,053 - mmdet - INFO - Epoch [1][7/27]        lr: 1.707e-04, eta: 0:06:52, time: 0.910, data_time: 0.062, memory: 7264, loss_heatmap: 6989.6382, layer_-1_loss_cls: 42.0245, layer_-1_loss_bbox: 8.0230, matched_ious: 0.0054, loss_l1: 13.2696, loss_l2: 222.9359, loss_absrel: 0.7619, loss: 7276.6528, grad_norm: 31704.3477
Traceback (most recent call last):
  File "./tools/train.py", line 269, in <module>
    main()
  File "./tools/train.py", line 258, in main
    train_model(
  File "/home/mmdet3d/apis/train.py", line 344, in train_model
    train_detector(
  File "/home/mmdet3d/apis/train.py", line 319, in train_detector
    runner.run(data_loaders, cfg.workflow)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 136, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 53, in train
    self.run_iter(data_batch, train_mode=True, **kwargs)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 31, in run_iter
    outputs = self.model.train_step(data_batch, self.optimizer,
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 63, in train_step
    output = self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/anaconda3/envs/lib/python3.8/site-packages/mmdet/models/detectors/base.py", line 248, in train_step
    losses = self(**data)
  File "/home/anaconda3/envs/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/mmdet3d/models/detectors/***.py", line 74, in forward
    return self.forward_train(**kwargs)
  File "/home/mmdet3d/models/detectors/***.py", line 250, in forward_train
    losses_occ = self.forward_train(
  File "/home/mmdet3d/models/dense_heads/***.py", line 400, in forward_train
    invalid = torch.isnan(grad_sigma)
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2861194) of binary: /home/anaconda3/envs/bin/python

tarashakhurana / 4d-occ-forecasting

CUDA error: invalid configuration argument #4