Code for point cloud segmentation visualization

iwajef commented 5 months ago

Hi authors. Thank you for the amazing work.

Is there a python script for visualizing the point cloud semantic segmentation result?

After training the model on SemanticKITTI or nuScenes dataset, the trained model is saved (e.g. "best_miou_iter_6000.pth"), and I want to use this model parameter file to visualize the point cloud segmentation result on the validation set of SemanticKITTI / nuScenes dataset for better understanding and comparisons.

ldkong1205 commented 5 months ago

Hi @iwajef, thanks for your interest in our work!

I can provide you a script for drawing the visualizations shown in the paper later. For now, you can refer to the visualization tool in MMDetection3D: https://mmdetection3d.readthedocs.io/en/latest/user_guides/visualization.html#drawing-3d-semantic-mask

You can also take a look at the visualization interface from Open3D-ML: https://github.com/isl-org/Open3D-ML

iwajef commented 5 months ago

I can provide you a script for drawing the visualizations shown in the paper later. For now, you can refer to the visualization tool in MMDetection3D: https://mmdetection3d.readthedocs.io/en/latest/user_guides/visualization.html#drawing-3d-semantic-mask

Thank you for replying! I tried to follow the visualization guideline in MMDection3D, with the command: python tools/test.py configs/lasermix/lasermix_cy3d_semi_nuscenes_10.py work_dirs/lasermix_cy3d_semi_nuscenes_10/best_miou_iter_6000.pth --show --show-dir ./visualize_result --task lidar_seg

But it exited with some error after showing first few frames as follows:

06/16 21:20:17 - mmengine - WARNING - The prefix is not set in metric class SegMetric. Loads checkpoint by local backend from path: work_dirs/lasermix_cy3d_semi_nuscenes_10/best_miou_iter_6000.pth 06/16 21:20:21 - mmengine - INFO - Load checkpoint from work_dirs/lasermix_cy3d_semi_nuscenes_10/best_miou_iter_6000.pth [Open3D WARNING] [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match. [Open3D WARNING] [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match. [Open3D WARNING] [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match. [Open3D WARNING] [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match. [Open3D WARNING] [ViewControl] ConvertFromPinholeCameraParameters() failed because window height and width do not match. Traceback (most recent call last): File "tools/test.py", line 162, in main() File "tools/test.py", line 158, in main runner.test() File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1823, in test metrics = self.test_loop.run() # type: ignore File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/loops.py", line 445, in run self.run_iter(idx, data_batch) File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(args, kwargs) File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/loops.py", line 466, in run_iter self.runner.call_hook( File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1839, in call_hook getattr(hook, fn_name)(self, kwargs) File "/home/user/code/gits/1_sota/LaserMix/mmdet3d/engine/hooks/visualization_hook.py", line 228, in after_test_iter self._visualizer.add_datasample( File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/dist/utils.py", line 427, in wrapper return func(args, **kwargs) File "/home/user/code/gits/1_sota/LaserMix/mmdet3d/visualization/local_visualizer.py", line 1042, in add_datasample self._draw_pts_sem_seg(data_input['points'], File "/home/user/code/gits/1_sota/LaserMix/mmdet3d/visualization/local_visualizer.py", line 771, in _draw_pts_sem_seg pts_color = palette[pts_sem_seg] IndexError: index 16 is out of bounds for axis 0 with size 16`

And when trying to visualize on SemanticKITTI, I got another error:

06/16 21:34:24 - mmengine - WARNING - The prefix is not set in metric class SegMetric. Loads checkpoint by local backend from path: work_dirs/lasermix_cy3d_semi_semantickitti_10/best_miou_iter_3000.pth 06/16 21:34:27 - mmengine - INFO - Load checkpoint from work_dirs/lasermix_cy3d_semi_semantickitti_10/best_miou_iter_3000.pth Traceback (most recent call last): File "tools/test.py", line 162, in main() File "tools/test.py", line 158, in main runner.test() File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1823, in test metrics = self.test_loop.run() # type: ignore File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/loops.py", line 445, in run self.run_iter(idx, data_batch) File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(args, kwargs) File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/loops.py", line 466, in run_iter self.runner.call_hook( File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1839, in call_hook getattr(hook, fn_name)(self, kwargs) File "/home/user/code/gits/1_sota/LaserMix/mmdet3d/engine/hooks/visualization_hook.py", line 228, in after_test_iter self._visualizer.add_datasample( File "/home/user/anaconda3/envs/lasermix/lib/python3.8/site-packages/mmengine/dist/utils.py", line 427, in wrapper return func(args, **kwargs) File "/home/user/code/gits/1_sota/LaserMix/mmdet3d/visualization/local_visualizer.py", line 980, in add_datasample keep_index = data_sample.gt_pts_seg.pts_semantic_mask != ignore_index # noqa: E501 AttributeError: 'PointData' object has no attribute 'pts_semantic_mask'

I think I might be missing some details to visualize the results correctly and I'll try to fix the errors. Anyway thanks for the suggestions and I'm looking forward to the visualization script :)

Odelllll commented 1 month ago

Hi, I encountered a similar problem when I followed the visualization guide in MMDection3D to try to visualize on the semantickitti dataset: AttributeError: 'PointData' object has no attribute 'pts_semantic_mask', What should I do to deal this problem?

Thank you very much.

ldkong1205 commented 1 month ago

Hi @Odelllll, for the mentioned problem, you can try to resolve it by checking:

Whether you are using the provided .pkl files? If not, you can download them from here.

Whether you have added the following in your config file:

dict(type='LoadAnnotations3D',
 with_bbox_3d=False, with_label_3d=False, with_seg_3d=True, seg_3d_dtype='np.uint8', backend_args=backend_args
),

If the above two do not solve your problem, I recommend you consider using the following codebase for LiDAR visualization: https://github.com/PRBonn/lidar-visualizer

Odelllll commented 1 month ago

Thank you very much for your suggestion, I will try it!

ldkong1205 / LaserMix

Code for point cloud segmentation visualization #30