plane x cm above ground v0

[melisandeteng]

The goal is to get a first baseline on how to make a mask for a plane Y = x cm (vertical axis) in a GSV image. Steps :

• get depth map associated with the image with Megadepth
• recover 3D relative coordinates (scale factor will be unknown)
• detect objects (vehicles) in the image, regress 3D bbox to get metric dimensions and orientation
• segment the vehicles
• for each segmented vehicle: match y_max - y_min in depthmap coordinates to the metric prediction of height of the vehicle
• do a weighted average on the vehicles to find Y threshold corresponding to x cm

-> we assume here that objects are all ON the ground

[melisandeteng]

Ideas to improve the pipeline:

• Infer the depthmap with MegaDepth on augmented data (average Megadepth output over image with transforms (flip, brightness, contrast) for a smoother output maybe ? )
• weight the 3D relative coordinates <-> metric system match with YOLO confidence

[melisandeteng]

How can we have better classes matching ?

• YOLO is used to detect the object with KITTI classes : ['cyclist', 'van', 'misc', 'tram', 'car', 'person_sitting', 'pedestrian', 'truck']
  • segmentation trained on cityscapes is used to get which pixels in a 2D bbox are actually of interest. The idea was also to filter out YOLO errors by keeping the bounding boxes only if a certain percentage of the pixels in it had the same segmentation label as the 2D detection label. However very few classes (labels) overlap: we can make 'person_sitting' and 'pedestrian' correspond to 'person' (segmentation label), but for 'cyclist' for example, the labels that we have for segmentation are 'rider' and 'bicycle' but this match is not obvious because there are also motorcycles and there could be bicycles without riders.

[vict0rsch]

should we close this @melisandeteng ? (if so, please do :) )