search

monniert / differentiable-blocksworld

[NeurIPS 2023] Code for "Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives"
https://www.tmonnier.com/DBW
MIT License
197 stars 10 forks source link

DTU Evaluation Metric #11

Closed Jiangshuyi0V0 closed 1 week ago

Jiangshuyi0V0 commented 6 months ago

Thank you for your awesome work. I have a few questions about the DTU-CD evaluation step.

I noticed that the evaluated distance in the code is based on the ground truth point cloud file with reconstructed blocks, excluding the background and floor, while the GT point cloud file includes the background. Could you kindly clarify whether you exclusively use the mean_d2s metric in your paper, or do you take the average of mean_d2s and means2d? If it's the latter, how do you handle the environmental point cloud in the referenced ground truth file?

monniert commented 5 months ago

Hi @Jiangshuyi0V0, sorry for the late reply, I was afk for a few weeks!

We use the standard DTU evaluation which evaluates a given mesh wrt to the GT point cloud files: you can find a python implementation in this codebase, which is a copy from this repo https://github.com/jzhangbs/DTUeval-python. Note that the GT point cloud indeed includes part of the background, but the evaluation script uses object masks (dilated by a few pixels IIRC, more information in the DTU paper: https://roboimagedata2.compute.dtu.dk/data/text/multiViewCVPR2014.pdf) to filter out GT points that do not belong to the object. Therefore, removing the background and floor from our predictions was more aligned with this evaluation process.

We report the average of mean_d2s and mean_s2d