search

waymo-research / waymo-open-dataset

Waymo Open Dataset
https://www.waymo.com/open
Other
2.65k stars 606 forks source link

Temporally Inconsistent 2D Video Segmentation Labels #588

Open nikhilgosala opened 1 year ago

nikhilgosala commented 1 year ago

There seems to be severe inconsistencies between semantic annotations of consecutive frames in the v1.4.0 and v1.4.1 releases. Additionally, there are also many mislabelled elements in the scene, for example, (1) cars being annotated as roads, (2) sidewalk being annotated as lane markers and road etc.

Could you please check the labels and confirm these label inconsistencies?

Some of the files with the above mentioned problems are:

  1. Timestamp: 1506904096471880, 1506904096671918 | Location: PHX | Scene Name: 15943938987133888575_2767_300_2787_300 1506904096471880-location_phx-15943938987133888575_2767_300_2787_300 1506904096671918-location_phx-15943938987133888575_2767_300_2787_300

  2. Timestamp: 1506904106885191, 1506904107285557 | Location: PHX | Scene Name: 15943938987133888575_2767_300_2787_300 1506904106885191-location_phx-15943938987133888575_2767_300_2787_300 1506904107285557-location_phx-15943938987133888575_2767_300_2787_300

  3. Timestamp: 1506907465550902, 1506907465950812 | Location: PHX | Scene Name: 14940138913070850675_5755_330_5775_330 1506907465550902-location_phx-14940138913070850675_5755_330_5775_330 1506907465950812-location_phx-14940138913070850675_5755_330_5775_330

alexzzhu commented 1 year ago

Hi, thanks for pointing this out! The first two pairs of frames are somewhat expected. The newest labels in the Perception dataset v1.4.1 release provide improved quality for 2D segmentation labels to better aid the research community. These changes improved semantic segmentation temporal consistency, but focused on classes with instances, where inconsistencies had a larger impact on tracking metrics. As a result, some small inconsistencies may remain for non-instance’d classes, but should not significantly affect current semantic segmentation metrics, which are largely per frame.

I checked the third frame and have identified the issue. I will put it on our list of TODOs, but I can't give a timeline of when this will be fixed. This issue should be relatively rare, and hopefully won't affect training.