search

isl-org / Open3D-ML

An extension of Open3D to address 3D Machine Learning tasks
Other
1.85k stars 316 forks source link

data.to(self.device) AttributeError: 'dict' object has no attribute 'to' #283

Closed yucedagonurcan closed 3 years ago

yucedagonurcan commented 3 years ago

Hello All,

I am in 'dev' branch in Open3D-ML and 'master' branch in Open3D.

I wanted to try KITTI inference with PointPillars with this script:

PointPillarsKITTI

It is surely rewrite of the example inference for Object Detection in README.

Here is the error I am getting with this code:

(3d_det) ➜ examples git:(dev) ✗ python kitti_point_pillars.py

Using the Open3D PyTorch ops with CUDA 11 may have stability issues!

We recommend to compile PyTorch from source with compile flags '-Xcompiler -fno-gnu-unique'

or use the PyTorch wheels at https://github.com/intel-isl/open3d_downloads/releases/tag/torch1.7.1

Ignore this message if PyTorch has been compiled with the aforementioned flags.

See https://github.com/intel-isl/Open3D/issues/3324 and https://github.com/pytorch/pytorch/issues/52663 for more information on this problem.

INFO - 2021-05-25 18:14:00,578 - object_detection - Loading checkpoint ./logs/pointpillars_kitti_202012221652utc.pth INFO - 2021-05-25 18:14:00,620 - kitti - Found 7518 pointclouds for test Traceback (most recent call last): File "kitti_point_pillars.py", line 30, in result = pipeline.run_inference(data) File "/home/onur/miniconda3/envs/3d_det/lib/python3.6/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 62, in run_inference data.to(self.device) AttributeError: 'dict' object has no attribute 'to'

_- Okay, I commented out the result = pipeline.run_inference(data) line and tried again since run_test function will create ConcatBatcher and then moves the object to the cuda (runinference gets a dictionary and tries to move it to cuda that's why the earlier error occurs.). I thought it would be fine but there is another error that I am not quite sure if it is about Open3D or torch or my CUDA:

(3d_det) ➜ examples git:(dev) ✗ python kitti_point_pillars.py

Using the Open3D PyTorch ops with CUDA 11 may have stability issues!

We recommend to compile PyTorch from source with compile flags '-Xcompiler -fno-gnu-unique'

or use the PyTorch wheels at https://github.com/intel-isl/open3d_downloads/releases/tag/torch1.7.1

Ignore this message if PyTorch has been compiled with the aforementioned flags.

See https://github.com/intel-isl/Open3D/issues/3324 and https://github.com/pytorch/pytorch/issues/52663 for more information on this problem.

INFO - 2021-05-25 18:21:21,330 - object_detection - Loading checkpoint ./logs/pointpillars_kitti_202012221652utc.pth INFO - 2021-05-25 18:21:21,375 - kitti - Found 7518 pointclouds for test INFO - 2021-05-25 18:21:21,381 - object_detection - DEVICE : cuda INFO - 2021-05-25 18:21:21,381 - object_detection - Logging in file : ./logs/PointPillars_KITTI_torch/log_test_2021-05-25_18:21:21.txt INFO - 2021-05-25 18:21:21,381 - kitti - Found 7518 pointclouds for test INFO - 2021-05-25 18:21:21,382 - object_detection - Initializing from scratch. INFO - 2021-05-25 18:21:21,382 - object_detection - DEVICE : cuda INFO - 2021-05-25 18:21:21,382 - object_detection - Logging in file : ./logs/PointPillars_KITTI_torch/log_valid_2021-05-25_18:21:21.txt INFO - 2021-05-25 18:21:21,382 - kitti - Found 3769 pointclouds for validation INFO - 2021-05-25 18:21:21,382 - object_detection - Started validation validation: 0%| | 0/3769 [00:00<?, ?it/s]/pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [0,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [1,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [2,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [3,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [4,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [5,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [6,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [7,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [8,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [9,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [10,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [11,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [13,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [15,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [17,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [18,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [19,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [20,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [21,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [22,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [23,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [24,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [25,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [26,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [27,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [28,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [29,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [30,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. /pytorch/aten/src/ATen/native/cuda/IndexKernel.cu:142: operator(): block: [0,0,0], thread: [31,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed. validation: 0%| | 0/3769 [00:00<?, ?it/s] Traceback (most recent call last): File "kitti_point_pillars.py", line 33, in pipeline.run_test() File "/home/onur/miniconda3/envs/3d_det/lib/python3.6/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 107, in run_test self.run_valid() File "/home/onur/miniconda3/envs/3d_det/lib/python3.6/site-packages/open3d/_ml3d/torch/pipelines/object_detection.py", line 165, in run_valid results = model(data) File "/home/onur/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, *kwargs) File "/home/onur/miniconda3/envs/3d_det/lib/python3.6/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 133, in forward x = self.extract_feats(inputs) File "/home/onur/miniconda3/envs/3d_det/lib/python3.6/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 108, in extract_feats x = self.middle_encoder(voxel_features, coors, batch_size) File "/home/onur/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, **kwargs) File "/home/onur/miniconda3/envs/3d_det/lib/python3.6/site-packages/open3d/_ml3d/torch/models/point_pillars.py", line 619, in forward voxels = voxel_features[batch_mask, :] RuntimeError: CUDA error: device-side assert triggered

I am quite beginner to codebase as you can understand. What am I missing here?

Have a great day.

yucedagonurcan commented 3 years ago

Fixed with new version

ghost commented 2 years ago

Hi, we are having the same issue as your last error message. Could you please clarify what new version and steps helped you fix it?