search

autonomousvision / differentiable_volumetric_rendering

This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"
http://www.cvlibs.net/publications/Niemeyer2020CVPR.pdf
MIT License
794 stars 90 forks source link

Evaluation scores very low #66

Closed Xinyu-Kang closed 2 years ago

Xinyu-Kang commented 2 years ago

Hi,

I ran 

python generate.py configs/single_view_reconstruction/multi_view_supervision/ours_combined_pretrained.yaml

to generate the 3D meshes and then ran

python eval_meshes.py configs/single_view_reconstruction/multi_view_supervision/ours_combined_pretrained.yaml

to do the evaluation. However, the output seems not to match the scores in the paper: <html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">

class name | completeness (mesh) | accuracy (mesh) | normals completeness (mesh) | normals accuracy (mesh) | normals (mesh) | completeness2 (mesh) | accuracy2 (mesh) | chamfer (mesh) | chamferL1 (mesh) -- | -- | -- | -- | -- | -- | -- | -- | -- | -- airplane | 0.014681822 | 0.017131265 | 0.781481175 | 0.817698949 | 0.799590062 | 0.000627151 | 0.00081252 | 0.000719836 | 0.015906543 bench | 0.017600321 | 0.020100394 | 0.685943895 | 0.712539858 | 0.699241876 | 0.001260887 | 0.001365951 | 0.001313419 | 0.018850358 cabinet | 0.021391467 | 0.015055348 | 0.756380033 | 0.838302329 | 0.797341181 | 0.001248461 | 0.00071197 | 0.000980216 | 0.018223407 car | 0.025601329 | 0.013600567 | 0.679889168 | 0.820496148 | 0.750192658 | 0.001295978 | 0.000414793 | 0.000855385 | 0.019600948 chair | 0.025060366 | 0.024991364 | 0.768567459 | 0.767983969 | 0.768275714 | 0.001634166 | 0.001505125 | 0.001569645 | 0.025025865 display | 0.027552643 | 0.026911068 | 0.829833097 | 0.823852186 | 0.826842642 | 0.002578801 | 0.00219671 | 0.002387756 | 0.027231856 lamp | 0.043013242 | 0.034546767 | 0.726022896 | 0.70846303 | 0.717242963 | 0.007536651 | 0.003446865 | 0.005491758 | 0.038780004 loudspeaker | 0.033482416 | 0.027726269 | 0.777037229 | 0.821831801 | 0.799434515 | 0.003024743 | 0.002342595 | 0.002683669 | 0.030604342 rifle | 0.013271801 | 0.018142186 | 0.641598026 | 0.581900868 | 0.611749447 | 0.000531731 | 0.001019649 | 0.00077569 | 0.015706993 sofa | 0.02633354 | 0.021773386 | 0.769530807 | 0.821057323 | 0.795294065 | 0.001617846 | 0.001385877 | 0.001501861 | 0.024053463 table | 0.020720319 | 0.022446619 | 0.806766468 | 0.802003734 | 0.804385101 | 0.003645217 | 0.003545516 | 0.005363192 | 0.020574939 telephone | 0.014733233 | 0.014831776 | 0.889298158 | 0.902264363 | 0.89578126 | 0.000984927 | 0.001199974 | 0.001092451 | 0.014782504 vessel | 0.028300423 | 0.020245 | 0.689226388 | 0.75181922 | 0.720522804 | 0.003560484 | 0.001115181 | 0.002337833 | 0.024272712

I was wondering if I need to scale/rotate the meshes somehow before evaluating them, or are there any potential reasons causing this problem?

Thank you!!

m-niemeyer commented 2 years ago

Please see previous issues like Issue 10: We follow previous works to use "use 1/10 times the maximal edge length of the current object’s bounding box as unit 1" (see Section 4 - Metrics). In practice, this means that you need to multiply e.g. the Chamfer-L1 metric by 10 to obtain the numbers you can find in the paper.