search

isl-org / TanksAndTemples

Toolbox for the TanksAndTemples benchmark website
Other
86 stars 23 forks source link

Self occlusion in ground truth #26

Closed raphaelsulzer closed 4 years ago

raphaelsulzer commented 4 years ago

Hi, and thanks for the great dataset.

I noticed that in some scenes of the training dataset there is a significant amount of self occlusion in the LiDAR ground truth (e.g. the chest in Ignatius, or the load floor on the truck. Some of these areas are however visible in (potentially overlapping) images. If I understand the evaluation procedure correctly, if points are reconstructed in these areas, it will drive down the F-Score.

Additionally, I noticed that in the Truck scene, there are some sort of fairy lights above the truck that are also not present in the ground truth. Since the bounding volume for clipping is only 2D (a polygon), this area is still evaluated in the reconstruction.

Are my assumptions correct and do these problems also occur in the test datasets (where we do not have the ground truth)?

arknapit commented 4 years ago

Hello Raphael, Yes, you noticed alright that there is self-occlusions/incompleteness in the GT files. We tried to have laser-scanner positions close to the camera path, so they should see the same surface and also the same occlusions. Since there are much less laser-scanner positions than camera-positions, this is off course not always possible. Additionally, if you have methods, which computes and closes surfaces (eg. meshing and sampling) you could also end up at a similar situation. We therefore selected and cropped the test-set groundtruth-scans as carefully as possible, but a small amount of such "non-agreeing" areas will off course always remain. However, these areas are quite small compared to the rest of the model, and the metric we use is quite robust against outliers. As a test, I also cropped the height of the Truck model in the evaluation of a COLMAP reconstruction, (to avoid penalizing for the correctly reconstructed "fairy lights" on top), and couldn't see significant changes in the P/R/F1 scores. But yes, there might be corner cases (e.g. very sparse reconstructions, or samplings from meshed reconstructions, learned geometries), where the limitations of our model and evaluation method will need to be taken into closer consideration. greetings Arno

raphaelsulzer commented 4 years ago

I see, okay. Thank you for the quick response!