[Question] Parameters for benchmarking

[Ntweat]

Hi

I am trying to benchmark many sparse point tools such as openMVG, colmap, MVE, etc along with dense point tools such as openMVS, SMVS, CMVS/PMVS, etc

I was wondering what parameters can I use for the benchmarking for both.

[pmoulon]

From OpenMVG (SFM side), you can only play with the pipeline and the matches (feature extraction presets).

Pipeline

• Incremental
• Global

Feature preset (ComputeMatches):

• NORMAL/HIGH/ULTRA

[Ntweat]

Sorry.

I meant to ask parameters for benchmarking.. Like number of camera pose estimates, completeness of model etc

[pmoulon]

You can take a look to the paper that perform benchmarking task... https://github.com/openMVG/awesome_3DReconstruction_list#mvs---point-cloud---surface-accuracy

[Ntweat]

Thank you

[pmoulon]

SfM side:

• Number or camera
• position accuracy
• orientation accuracy (rotation)

MVS Side:

• surface completness
• proximity of the true surface
• ...

[Ntweat]

During a preliminary analysis, I am finding that incremental pipeline is giving more camera pose estimates than global.

Why is that?

Also, COLMAP is giving more accurate pose estimates than openMVG.
I have sent a dataset, it has both the above points

[pmoulon]

I invite you to read here about the difference between incremental SfM and Global SfM

Which process did you use to compare the pose accuracy? How much more accurate COLMAP is for your given dataset? Did you check that if the number of feature is similar? Are you sure you compare the SfM tools in the same settings (grouped vs. ungrouped intrinsics)?

• I mean having the intrinsic grouped or not can lead to some differences depending of the camera settings you used.

I see nothing after your comment:

I have sent a dataset, it has both the above points

[Ntweat]

I had sent a dataset on the email id you had given for https://github.com/openMVG/openMVG/issues/1149 (If you did not get it I will send it again)

All 3 pipelines had the same settings: (oepnMVG incremental, Global and COLMAP) SIFT Exhaustive Matcher (FLANN matcher) Ungrouped intrinsics

The accuracy can be seen in sparse point clouds produced. The dataset I sent is of my car, in that in incremental openmvg there is shift which produces a ghost windscreen during dense point reconstruction.

[pmoulon]

Did you check if the pipeline have more or less the same of features extracted per image?

Please remember that cars are hard object to do in photogrammetry since they are not lambertian (reflective object are hard to handle).

[Ntweat]

I used the same SIFT parameters for all pipelines. All of them gave more or less same features.

openMVG Global and Incremental had the same features as the next stage (compute_matches) the pipeline changes.

I know cars/vehicles are tough, I am very close to getting good results, with openMVG Incremental pipeline. The issues are only in few objects like the car I sent. Otherwise, openMVG Incremental is out performing COLMAP.