Inaccurate ground reconstruction in segment-1067626

[xslittlegrass]

I'm following the instruction to reprocess and train the model on session segment-10676267326664322837_311_180_331_180_with_camera_labels, with config withmask_nolidar.230814.yaml. However there are a couple of big holes in the reconstructed ground (see screenshots). The normal and depth priors all look normal though.

I'm wondering whether you have any insights on the cause of this issue?

[ventusff]

Yes, i have the same observation. This is because the monocular normals inferred at certain consecutive frames by Omnidata is severely incorrect: You can check it yourself by checking the tensorboard (the coll_normals_gt_pred entry) and the extracted normals .jpg files.

You may want to consider adjusting the weights, such as reducing the weight of 'mono_normals' by a factor of ten and increasing the weight of 'mono_depth' (especially mono_w_reg) by the same factor. It will work for this sequence, but will compromise other sequences as the inferred monocular depths cues can also be rough and incorrect :(

[xslittlegrass]

Thanks for the explanation! For the evaluation results in table 1 in the paper (Table 1: Quantitative comparison on selected Waymo Open Dataset sequences), did you hand tuned the weights (or other parameters) for each sequence or use the same parameters as in configs/waymo/streetsurf for all sequences?

[ventusff]

Under each setting, the same config and weights are applied to all sequences, which are also included in the config yamls released.