search

xiumingzhang / GenRe-ShapeHD

Code and Data Release for GenRe (NeurIPS 2018) and ShapeHD (ECCV 2018)
http://genre.csail.mit.edu/
294 stars 62 forks source link

Evaluation code #67

Open xianyongqin opened 4 years ago

xianyongqin commented 4 years ago

Thanks for releasing the amazing codes. Is it possible to release the evaluation codes that reproduce the Chamfer distance (GenRe, 0.106) reported in Table 1 of the GenRe paper? I followed the setting described in the paper but got a much worse Chamfer distance (GenRe, 0.30) with the released pre-trained model.

yuchenrao commented 3 years ago

Hi @xianyongqin, I also have worse evaluation by using their evaluation code from pix3d. For Lamp, the paper has 0.124 but mine is 0.297. Did you figure out why this happen? Thanks a lot!

ztzhang commented 3 years ago

referencing to issue #73. For CD discrepancies, several things might be helpful for tracing the issue:

  1. the gt voxel should be surface only.
  2. the prediction should at least match the given mask, at the test viewing angle.
  3. for results in table 1, we searched for different thresholding values, usually from 0.3 to 0.5 with a 0.05 step size, and this is done for each class. Though this is not ideal, I think we did this since previous baselines reported their numbers in a similar fashion.