Scannet test results

[mrharicot]

Hi,

Thanks for sharing the code for your work!

I have a few questions regarding the evaluation on the ScanNet dataset:

• How can I reproduce the results in Table 1 of the paper?
• How do you average the results?
  1. Do you compute the results per scannet scene, then average these across all scenes?
  2. Or do you bulk test across all groups of keyframes for the whole ScanNet test set and average per-frame ?

If the latter, could you please share the keyframe file used for the scannet test ?

Thank you for your help!

[ardaduz]

Hi,

Starting with your second question, we do the latter: average over a keyframe, average over all the keyframes in all the scenes.

To reproduce the results in Table 1 and for your convenience, I am providing a partially populated test setup with the folder structure and keyframe indices: https://drive.google.com/file/d/17vySxdAVOUIqevgNCtM5t-YeZRo2nc-H/view?usp=sharing

Keyframe indices are the ones that we use to acquire the results and they are created with simulate_keyframe_buffer.py. I placed only scene0707_00 to perhaps give you clues, you can populate the others.

Then, please have a look at https://github.com/ardaduz/deep-video-mvs#testing:

1. Set the input output locations in Config and disable visualization.
   • visualize=False
   • test_offline_data_path = "PATH_TO_DeepVideoMVS-Test-Setup"
   • test_dataset_name = "scannet"
   • test_result_folder = "PATH_TO_DeepVideoMVS-Test-Setup/results"
2. Run pairnet/run_testing.py or fusionnet/run_testing.py depending on which one you want to test
3. The depth predictions for the keyframes "predictions" and per-frame averaged metrics "errors" are saved in test_result_folder as .npz for each scene separately.
4. When finished, you can load each scene's errors one-by-one and concatenate all and average.

Hope this helps, let me know if you manage or encounter any issue.

[mrharicot]

Hi,

Thanks a lot for your answer! I'll try asap and let you know how it goes.

[mrharicot]

Hi,

Thank you for your help, I got it running and matched the results reported in your paper. Actually the delta < 1.25 is even very so slightly better at 96.59 (vs 96.54 reported on arxiv).