search

GhiXu / ACMP

Planar Prior Assisted PatchMatch Multi-View Stereo
MIT License
188 stars 30 forks source link

Unable to reproduce ETH3D results #5

Closed drkoller closed 3 years ago

drkoller commented 3 years ago

Thank you for making your code available!

I am trying to use your ACMP code to reproduce your reconstruction results that are posted on the ETH3D website and that were published in your AAAI-20 paper. My methodology is:

  1. Build the ACMP code from this repository (including PR #4)
  2. Download the High-res multi-view training datasets (multi_view_training_dslr_undistorted.7z) from the ETH3D website
  3. Use colmap2mvsnet_acm.py with default parameters to convert the COLMAP files to MVSNet format
  4. Run the ACMP executable to reconstruct 3D point clouds
  5. Run ETH3DMultiViewEvaluation to compute the F1 score at 2 cm tolerance

When I use your code to reconstruct and evaluate the point clouds in this manner, I get some F1 scores that differ significantly from those posted on the ETH3D website:

Dataset F1 score
(GitHub
code)		 F1 score
(ETH3D
website)		 Δ
courtyard 84.09 85.61 -1.52
delivery 85.51 85.68 -0.17
electro 86.23 85.61 +0.62
facade 69.06 69.59 -0.53
kicker 72.83 77.33 -4.50
meadow 60.53 71.00 -10.47
office 74.17 74.45 -0.28
pipes 68.95 69.16 -0.21
playground 69.23 71.88 -2.65
relief 84.74 83.15 +1.59
relief_2 84.21 83.37 +0.84
terrace 90.19 89.92 +0.27
terrains 88.67 90.57 -1.90

I realize that the algorithm is non-deterministic, and that F1 scores may vary slightly from one execution to the next. However, the F1 scores for the kicker and meadow datasets are much worse when I run this ACMP code, compared to the results posted on the ETH3D website.

What are the differences between the code posted in this repository and the code that was used to generate the results in the paper and posted on the ETH3D website?

blablebli-robots commented 3 years ago

@drkoller could you please share your pair.txt files for the kicker and meadow sets? I cannot reproduce the results either.

Thanks!

drkoller commented 3 years ago

@blablebli-robots, here are my pair.txt files:

kicker-pair.txt meadow-pair.txt

Do your F1 scores match mine closely?

opaezosuna commented 3 years ago

@drkoller Thank you very much for sharing your files. My values are all over the place, I am getting just an unorganized point cloud in the final ply file. I tried from scratch with my own images and manually editing a pair.txt file, but again getting bad results.

GhiXu commented 3 years ago

Thanks for your feedback. The code is updated! I retest the code on high-res ETH3D training dataset and get almost the same results as those posted on the ETH3D website.