Why my testing mean CD of background reconstruction is 0.279369?

[ThePassedWind]

After running python main.py --mode test --config ./configs/test_bg_PIFu.yaml, I only got 1135 bg mesh in ./checkpoints/bgPIFu_result_1205; Then, run python evaluate_bg.py --gt_dir ./data/3dfront/bgdepth --result_dir ./checkpoints/bgPIFu_result_1205 and got mean CD of bg reconstruction 0.279369, which is better than that of your paper.

[HaolinLiu97]

Hi, did you specify the test_id in FRONT_bg_dataset class? The number does not match because it has not tested the whole test dataset. You can print the self.split in the FRONT_bg_dataset to see why the testing program only output one result.

[ThePassedWind]

I got it, thanks!

[ThePassedWind]

Additionally, I want to know how many scenes do you use for testing background reconstruction (3D-FRONT dataset). There are many "rendertaskxxxx" (not exist in './data/3dfront/prepare_data/xxx.pkl').

I just find 5167/8945 valid scenes to test bg reconstruction.

[HaolinLiu97]

@ThePassedWind Hi I will check it later. It should be the minority that the prepare data does not exist.

[ThePassedWind]

@HaolinLiu97 Yes, I think testing set of prepare data may be not complete. Reading top 2000 rendertasks of InstPIFu/data/3dfront/bg_split/test.json. Only 1135/2000 exist, non_exist scene id as below: not_exist_list.json

Please help checking this, thank you very much!

[ThePassedWind]

I tried to test 5167/8945 valid scenes, the mean CD is:

[HaolinLiu97]

@ThePassedWind this 5167 scenes may contain training scenes. I think the cd should be valid if you test the available 1135 test scene. I am stilling downloading the data to see why there is some missing samples.

[ThePassedWind]

Do your scene ids of testing set for bg reconstruction are between [3000,9000]? (like object reconstruction) I find there are many ids in InstPIFu/data/3dfront/bg_split/test.json out of [3000,9000].

[HaolinLiu97]

@ThePassedWind I think it should be fine if the test ids is in [0,9000], other than these are the training data.

[ThePassedWind]

Ohh, in this case, 5167/8945 scenes in InstPIFu/data/3dfront/bg_split/test.json would be valid (in [0,9000]). But I don't know why mean CD is so low.

[HaolinLiu97]

@ThePassedWind I need some time to setup the experiment and evaluation again. If the background mesh result looks fine, then the CD should be fine. Possibly I have updated some codes for the training causing this decrease, I will need to check it later.

[ThePassedWind]

@HaolinLiu97 Thanks for your help!

[ThePassedWind]

And I want to ask how do you evaluate full scene? Directly combine foreground instances(use 3D object boxes predicted by Im3D) and background meshes to be one mesh, and sample 10K points from pred mesh and GT mesh to compute CD and fscore?

[HaolinLiu97]

@ThePassedWind Yes it is

[ThePassedWind]

In this case, did you use ICP to align the whole mesh to the whole GT mesh, then sample 10K points to compute metrics? And if need to normalize pred mesh and gt mesh to [-1,1]?

And I want to ask how do you evaluate full scene? Directly combine foreground instances(use 3D object boxes predicted by Im3D) and background meshes to be one mesh, and sample 10K points from pred mesh and GT mesh to compute CD and fscore?

[HaolinLiu97]

@ThePassedWind I really do not remember that many details from two years ago. But I think it is reasonable to use the ICP alignment first. But you can also try not to use ICP to see if the metric is close to that in the paper. And I think the mesh normalization is not used, it is quite similar as evaluating the background CD.

[ThePassedWind]

@ThePassedWind I really do not remember that many details from two years ago. But I think it is reasonable to use the ICP alignment first. But you can also try not to use ICP to see if the metric is close to that in the paper. And I think the mesh normalization is not used, it is quite similar as evaluating the background CD.

Thanks again!