Occupancy Prediction visualization

[OliverZhu4]

Really appreciate your time and patience in answering the questions. I'm wondering if there's any way to visualize the result of occupancy prediction results. Current visualization methods only support bounding box visualizations, not occupancy predictions and I'm not sure if I missed any parts that explain how it's visualized. Thank you for your time!

[ArdrieuxPan]

I got same problem and wish it would be fix soon.

take the visualization for official pretrained model for example: running command ./tools/dist_test.sh configs/bevdet_occ/bevdet-occ-stbase-4d-stereo-512x1408-24e.py ckpts/bevdet-occ-stbase-4d-stereo-512x1408-24e.pth 2 --eval mAP on console navigated to path ~/BEVdet, it would generate result as follow:

100%|████████████████████████████████████████████████████████████████████████████████| 162/162 [00:00<00:00, 175.77it/s]
===> per class IoU of 162 samples:
===> others - IoU = 40.28
===> barrier - IoU = 59.85
===> bicycle - IoU = 26.48
===> bus - IoU = 44.8
===> car - IoU = 50.91
===> construction_vehicle - IoU = 0.0
===> motorcycle - IoU = 23.08
===> pedestrian - IoU = 31.17
===> traffic_cone - IoU = 8.73
===> trailer - IoU = 46.55
===> truck - IoU = 41.89
===> driveable_surface - IoU = 80.8
===> other_flat - IoU = 56.4
===> sidewalk - IoU = 53.58
===> terrain - IoU = 53.36
===> manmade - IoU = 53.45
===> vegetation - IoU = 46.28
===> mIoU of 162 samples: 42.21
(['others', 'barrier', 'bicycle', 'bus', 'car', 'construction_vehicle', 'motorcycle', 'pedestrian', 'traffic_cone', 'trailer', 'truck', 'driveable_surface', 'other_flat', 'sidewalk', 'terrain', 'manmade', 'vegetation', 'free'], array([0.40275752, 0.59849341, 0.26480666, 0.44795608, 0.50914966,
       0.        , 0.23082011, 0.31172203, 0.08732877, 0.46552761,
       0.41885617, 0.80796465, 0.5639872 , 0.53584636, 0.53361193,
       0.53445372, 0.46284511, 0.91786375]), 162)

but when executing command python tools/test.py configs/bevdet_occ/bevdet-occ-stbase-4d-stereo-512x1408-24e.py ckpts/bevdet-occ-stbase-4d-stereo-512x1408-24e.pth --format-only --eval-options jsonfile_prefix=outputs here comes the error:

[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 162/162, 0.1 task/s, elapsed: 1142s, ETA:     0s/home/apan/BEVDet/mmdet3d/datasets/nuscenes_dataset.py:522: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  if not ('pts_bbox' in results[0] or 'img_bbox' in results[0]):
Start to convert detection format...
[                                                  ] 0/162, elapsed: 0s, ETA:Traceback (most recent call last):
  File "tools/test.py", line 267, in <module>
    main()
  File "tools/test.py", line 253, in main
    dataset.format_results(outputs, **kwargs)
  File "/home/apan/BEVDet/mmdet3d/datasets/nuscenes_dataset.py", line 523, in format_results
    result_files = self._format_bbox(results, jsonfile_prefix)
  File "/home/apan/BEVDet/mmdet3d/datasets/nuscenes_dataset.py", line 362, in _format_bbox
    boxes = det['boxes_3d'].tensor.numpy()
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices

[Rayn-Wu]

Hi, I've encountered a similar situation before. Here is a solution that can help： in the file "mmdet3d/datasets/nuscenes_dataset_occ.py" line 86 you can remove the 'if' condition and set 'show_dir' the way you want:

            if index%100==0 and show_dir is not None:
                gt_vis = self.vis_occ(gt_semantics)
                pred_vis = self.vis_occ(occ_pred)
                mmcv.imwrite(np.concatenate([gt_vis, pred_vis], axis=1),
                             os.path.join(show_dir + "%d.jpg"%index))

by doing so, you can get an occ image like this:

I'm not sure if there are better methods to achieve this functionality. If there are, we can discuss together. I hope this can be helpful to you :) By the way I find the colors of this occ image strange as well...