[WARNING] batch has 0 intersections!!
Process Process-3:
Traceback (most recent call last):
File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/wuyou/zbh/BundleSDF/bundlesdf.py", line 228, in run_nerf
nerf.train()
File "/home/wuyou/zbh/BundleSDF/nerf_runner.py", line 862, in train
self.train_loop(batch.cuda())
File "/home/wuyou/zbh/BundleSDF/nerf_runner.py", line 758, in train_loop
self.amp_scaler.scale(loss).backward()
File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward
Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
RuntimeError: CUDA error: invalid configuration argument
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[WARNING] batch has 0 intersections!! Process Process-3: Traceback (most recent call last): File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap self.run() File "/opt/conda/envs/py38/lib/python3.8/multiprocessing/process.py", line 108, in run self._target(*self._args, **self._kwargs) File "/home/wuyou/zbh/BundleSDF/bundlesdf.py", line 228, in run_nerf nerf.train() File "/home/wuyou/zbh/BundleSDF/nerf_runner.py", line 862, in train self.train_loop(batch.cuda()) File "/home/wuyou/zbh/BundleSDF/nerf_runner.py", line 758, in train_loop self.amp_scaler.scale(loss).backward() File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/_tensor.py", line 363, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/opt/conda/envs/py38/lib/python3.8/site-packages/torch/autograd/init.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA error: invalid configuration argument CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1.