search

ahayler / s4c

BSD 2-Clause "Simplified" License
53 stars 2 forks source link

How did you handle invalid masks while evaluating on SSCBench-KITTI360 #2

Closed haiphamcse closed 10 months ago

haiphamcse commented 10 months ago

Hi there, loved your work. I just loaded your preprocessed GT for KITTI360 and noticed that you're still using the invalid mask from SSC (which covers the entire skies), did you noticed weird artifacts for MonoScene while training with these masks?

image

ahayler commented 10 months ago

Hi @haiphamcse, thank you for interest in our work. S4C does not train on any 3D ground truth. The 3D data we provide is purely for evaluation purposes. Because of this, we use the data from the SSCBench paper, but correctly preprocess it to make it easier to work with (for us and other researchers). In particular, the invalid masks provided are exactly the same as in SSCBench. As you have realized, these masks are far from perfect. Therefore, we did add some additional invalids (below street level) in our evaluation script. We did not remove any invalids during evaluation. More details can be found in the additional material of the S4C paper. As SSCBench does unfortunately not provide any model checkpoints, we also had to try to reproduce their reported results. We managed to this for all but the VoxFormer paper, which we were therefore unable to report results for (as we have to run the evaluate the trained model with the additional invalids). We trained all these models with the correctly pre-processed 3D data (this does not include our additional invalids). During our limited qualitative evaluation of our results on MonoScene (you can see some examples in the video and paper plots), I did not see any weird artefacts (within the field of view of the camera). The quantitive evaluation also matched the results reported in the SSCBench paper.

haiphamcse commented 10 months ago

Hi there, thank you for the quick reply. So just to clarify, you still keep the original invalid masks from SSCBench while training 3D baselines (MonoScene, LMSCNet) but while evaluating you add an extra mask under the drivable surface?

ahayler commented 10 months ago

Exactly, we wanted to replicate the model checkpoints from SSCBench. For more details please refer to the additional material.