Missing 3d Bouding box?

[xixinzhang]

I have some questions about the labels of 3d BBox:

1. I found the start frame(2897) of boxes in sequences 4 does not match the start frame(0) of the LiDAR scan. Is this something expected?
2. I've also seen a similar issue https://github.com/autonomousvision/kitti360Scripts/issues/40, which is related to data of sequence 0. There seem to be a lot of missing bboxes, I haven't checked all frames of all sequences, but here are some examples below. I don't know if I missed some data or if they were just not given, but the case I give for sequence 4 is out of the range of the first 250 frames.

• sequence 2 (frame 4433 front view):

• sequence 4 (frame 2907 front view) (frame 3368 front view, no label at all) (frame 3368 back view, no label at all) Only three static cars are being labeled for the frame window [3356, 3586], which is wrong from the above visualization

  <object1100>
  <index>8576</index>
  <label>car</label>
  <semanticId_orig>7</semanticId_orig>
  <semanticId>13</semanticId>
  <instanceId>129</instanceId>
  <category>instance</category>
  <timestamp>-1</timestamp>
  <dynamic>0</dynamic>
  <transform type_id="opencv-matrix">
    <rows>4</rows>
    <cols>4</cols>
    <dt>f</dt>
    <data>
      1.14138556e+00 2.14470959e+00 -6.63366588e-03 3.82886426e+03
      -5.34604931e+00 4.57898617e-01 -1.12612871e-03 -1.66201196e+03
      9.46977234e-04 9.00010951e-03 1.63808441e+00 2.39120895e+02 0. 0.
      0. 1.</data></transform>
  <vertices type_id="opencv-matrix">
    <rows>8</rows>
    <cols>3</cols>
    <dt>f</dt>
    <data>
      5.00000000e-01 5.00000000e-01 5.00000000e-01 5.00000000e-01
      5.00000000e-01 -5.00000000e-01 5.00000000e-01 -5.00000000e-01
      5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      -5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      -5.00000000e-01 -5.00000000e-01 -5.00000000e-01 5.00000000e-01</data></vertices>
  <faces type_id="opencv-matrix">
    <rows>12</rows>
    <cols>3</cols>
    <dt>u</dt>
    <data>
      0 2 1 2 3 1 4 6 5 6 7 5 4 5 1 5 0 1 7 6 2 6 3 2 5 7 0 7 2 0 1 3 4
      3 6 4</data></faces>
  <start_frame>3356</start_frame>
  <end_frame>3586</end_frame></object1100>
  <object1101>
  <index>8577</index>
  <label>car</label>
  <semanticId_orig>7</semanticId_orig>
  <semanticId>13</semanticId>
  <instanceId>130</instanceId>
  <category>instance</category>
  <timestamp>-1</timestamp>
  <dynamic>0</dynamic>
  <transform type_id="opencv-matrix">
    <rows>4</rows>
    <cols>4</cols>
    <dt>f</dt>
    <data>
      4.78651315e-01 1.92826843e+00 7.02159759e-03 3.82994263e+03
      -4.01186228e+00 2.29451627e-01 -1.06386833e-01 -1.66836865e+03
      -3.10723543e-01 7.85248913e-03 1.38441491e+00 2.38654770e+02 0. 0.
      0. 1.</data></transform>
  <vertices type_id="opencv-matrix">
    <rows>8</rows>
    <cols>3</cols>
    <dt>f</dt>
    <data>
      5.00000000e-01 5.00000000e-01 5.00000000e-01 5.00000000e-01
      5.00000000e-01 -5.00000000e-01 5.00000000e-01 -5.00000000e-01
      5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      -5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      -5.00000000e-01 -5.00000000e-01 -5.00000000e-01 5.00000000e-01</data></vertices>
  <faces type_id="opencv-matrix">
    <rows>12</rows>
    <cols>3</cols>
    <dt>u</dt>
    <data>
      0 2 1 2 3 1 4 6 5 6 7 5 4 5 1 5 0 1 7 6 2 6 3 2 5 7 0 7 2 0 1 3 4
      3 6 4</data></faces>
  <start_frame>3356</start_frame>
  <end_frame>3586</end_frame></object1101>
  <object1103>
  <index>8579</index>
  <label>car</label>
  <semanticId_orig>7</semanticId_orig>
  <semanticId>13</semanticId>
  <instanceId>131</instanceId>
  <category>instance</category>
  <timestamp>-1</timestamp>
  <dynamic>0</dynamic>
  <transform type_id="opencv-matrix">
    <rows>4</rows>
    <cols>4</cols>
    <dt>f</dt>
    <data>
      -1.57342708e+00 -1.90071630e+00 -6.44791964e-03 3.83258594e+03
      4.33695078e+00 -6.89578295e-01 -1.15723989e-03 -1.65412012e+03
      -3.29058268e-03 -8.37895181e-03 1.55791259e+00 2.39629547e+02 0.
      0. 0. 1.</data></transform>
  <vertices type_id="opencv-matrix">
    <rows>8</rows>
    <cols>3</cols>
    <dt>f</dt>
    <data>
      5.00000000e-01 5.00000000e-01 5.00000000e-01 5.00000000e-01
      5.00000000e-01 -5.00000000e-01 5.00000000e-01 -5.00000000e-01
      5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      -5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      5.00000000e-01 5.00000000e-01 -5.00000000e-01 -5.00000000e-01
      -5.00000000e-01 -5.00000000e-01 -5.00000000e-01 5.00000000e-01</data></vertices>
  <faces type_id="opencv-matrix">
    <rows>12</rows>
    <cols>3</cols>
    <dt>u</dt>
    <data>
      0 2 1 2 3 1 4 6 5 6 7 5 4 5 1 5 0 1 7 6 2 6 3 2 5 7 0 7 2 0 1 3 4
      3 6 4</data></faces>
  <start_frame>3356</start_frame>
  <end_frame>3586</end_frame></object1103>

[yiyiliao]

Thank you for reporting the issues!

I found the start frame(2897) of boxes in sequences 4 does not match the start frame(0) of the LiDAR scan. Is this something expected?

Yes, it is expected. We have withheld some RGB images from sequence 4 because they are used as test sequences for the NVS benchmark.

I've also seen a similar issue https://github.com/autonomousvision/kitti360Scripts/issues/40, which is related to data of sequence 0. There seem to be a lot of missing bboxes, I haven't checked all frames of all sequences, but here are some examples below. I don't know if I missed some data or if they were just not given, but the case I give for sequence 4 is out of the range of the first 250 frames.

I investigated this issue and found that the missing bounding boxes are likely due to retrieving them using the annotation window. Some regions have multiple annotations because the ego-car passed the same areas repeatedly or because of overlaps between adjacent windows. To address this, we implemented an automatic check to filter out duplicate annotations, which might explain why some boxes appear from a different window. Below are screenshots showing the same sequence (seq 4, 0000003356_0000003586.ply) overlaid with instance bounding boxes, comparing retrieval using the annotation window versus retrieval based on distance."

1. Annotation window-based retrieval

2. Distance-based retrieval

[yiyiliao]

I have updated kitti360Viewer3D.py to visualize the bounding boxes based on distance instead of annotation window: https://github.com/autonomousvision/kitti360Scripts/blob/d4d4885102f72ea8d96c9e72e2ff03a8834353a4/kitti360scripts/viewer/kitti360Viewer3D.py#L319-L345