Benchmarking

[merkato]

It will be somehow practical to create some benchmarking - for example log file with execution time for toolchain elements. Right now these things are echoed to terminal - we only need some command line switch (eg. --time) and results file.

Right now I'm trying to compare ODM with Agisoft PhotoScan on my machine, and ODM worked about 1h20 min, Agisoft is working 2h 10 minutes right now, and probably next 2h for Dense Reconstruction stage ;) Stay tuned for results ;)

[Fi156]

Very cool though, I compared the results of these SW products but not the time needed for processing. Please be aware of the fact that Photoscan might utilize your GPU, as well as other things like reconstruction density... (Standard outcome for the geotiff was in my case 150mb (ODM) to approx. 1.5gb in Photoscan.)

[merkato]

On Boruszyn dataset, with same hardware specs (Lenovo T500 laptop), output was 496 MB Tiff (ODM), 52 MB (LZW, Photoscan). Execution times were: 1h20' for ODM, 5h15' for Photoscan. And results are comparable - holes in texturing are visible. On the other side, for my Koczy Zamek site dataset (46 photos + GPS), with old quarry, Photoscan is many times better, OpenSfM created unconected 3 reconstructions, and ortho for that was textured with sky from photos. In Photoscan Orthophoto was generated correctly. How can I help with development for making better results with mixed datasets (oblique + nadir, only oblique, only horizontal), and for more difficult terrain (hills, rocks, etc.)

[smathermather]

This is awesome, @merkato. When you run git rev-parse HEAD inside the ODM repo, what do you get in response?

What parameters, if any are you running ODM with?

[merkato]

~/dev/OpenDroneMap$ git rev-parse HEAD
3fcc58da7a88f1d2f21807f97395d78b29692ae5

Boruszyn dataset was reconstructed with default settings !!! :dart:

Keep calm and do better SfM ;)

[dakotabenjamin]

I added it to the roadmap for the codesprint

[dakotabenjamin]

@merkato if you wanna help with development, we are doing a code sprint this week. Feel free to join on the gitter chat

[dakotabenjamin]

I've been looking into profiling / benchmarking and found cProfile which profiles every single command, but this seems a little too verbose to be useful:

python -m cProfile -s cumtime run.py --project-path tests/test_data/ --rerun pmvs
...
         250266 function calls (248001 primitive calls) in 997.591 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.009    0.009  997.594  997.594 run.py:3(<module>)
        3    0.000    0.000  991.982  330.661 system.py:16(run)
        3  991.981  330.660  991.981  330.660 {posix.system}
        1    0.001    0.001  990.734  990.734 pmvs.py:35(process)
        1    0.010    0.010    3.798    3.798 dataset.py:21(process)
       57    0.035    0.001    3.787    0.066 types.py:16(__init__)
       57    0.020    0.000    3.748    0.066 types.py:51(parse_pyexiv2_values)
       57    3.280    0.058    3.280    0.058 {cv2.imread}
        1    0.007    0.007    2.485    2.485 odm_georeferencing.py:25(process)
        1    0.000    0.000    1.249    1.249 types.py:127(convert_to_las)
       57    0.017    0.000    1.228    0.022 types.py:146(utm_to_latlon)
       57    0.003    0.000    0.521    0.009 system.py:30(run_and_return)
       57    0.512    0.009    0.513    0.009 metadata.py:110(write)
      171    0.002    0.000    0.371    0.002 subprocess.py:473(_eintr_retry_call)
       57    0.003    0.000    0.360    0.006 subprocess.py:650(__init__)
       57    0.014    0.000    0.350    0.006 subprocess.py:1187(_execute_child)
        2    0.078    0.039    0.270    0.135 __init__.py:29(<module>)
        1    0.002    0.002    0.233    0.233 odm_app.py:1(<module>)
        1    0.054    0.054    0.230    0.230 types.py:1(<module>)
       57    0.220    0.004    0.220    0.004 {posix.read}
     3876    0.004    0.000    0.199    0.000 exif.py:188(_get_value)
     3876    0.012    0.000    0.195    0.000 exif.py:171(_compute_value)
     3876    0.016    0.000    0.193    0.000 metadata.py:196(__getitem__)
     3876    0.104    0.000    0.171    0.000 metadata.py:163(_get_exif_tag)
     4104    0.026    0.000    0.164    0.000 exif.py:257(_convert_to_python)
       57    0.002    0.000    0.156    0.003 subprocess.py:768(communicate)
      114    0.151    0.001    0.151    0.001 {method 'read' of 'file' objects}

So I'll write something more specific to this project. Something like for each part in the chain append the total runtime for that process to a benchmark file. Should be simple enough.

[dakotabenjamin]

@merkato do you think you could check out this PR? #258 has implemented a fairly simple benchmarking tool. Is there anything else you think I should add to it? EDIT: For your reference, this is the file it produces:

ODM Benchmarking file created Wed Feb 24 15:41:40  2016
Number of Cores: 32
 Resizing runtime: 0.001394 seconds
 OpenSfM runtime: 0.000214 seconds
 CMVS runtime: 0.000133 seconds
 PMVS runtime: 0.000111 seconds
 Meshing runtime: 0.000129 seconds
 Texturing runtime: 0.000129 seconds
 Georeferencing runtime: 2.216099 seconds

[smathermather]

+1

[merkato]

+1 I'll try on saturday morning. Right now I'm learning new things on railway. Till 10th of March There's no free time. @dakotabenjamin thank you for this PR.

[merkato]

I've cloned your cmake-benchmarks from your fork, and build it with no problems.

But... There's no effects :(

With this parameters run.py --time --project-path /home/mechanik/dev/odm_data/odm_boruszyn_kap/ --odm_georeferencing-gcpFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/gcp_list.txt --odm_georeferencing-useGcp project gets reconstructed, but georef is failed...

[INFO] Running OMD Georeferencing Cell
[DEBUG] running /home/mechanik/dev/OpenDroneMap/build/bin/odm_georef -bundleFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/opensfm/bundle_r000.out -inputCoordFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/coords.txt -inputFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_texturing/odm_textured_model.obj -outputFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/pmvs/recon0/models/option-0000.ply -outputPointCloudFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/odm_georeferenced_model.ply -logFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/odm_georeferencing_log.txt -georefFileOutputPath /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/odm_georeferencing_model_geo.txt
Error in Georef:
Failed opening coordinate file /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/coords.txt for reading.
quitting cause: 
    /home/mechanik/dev/OpenDroneMap/build/bin/odm_georef -bundleFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/opensfm/bundle_r000.out -inputCoordFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/coords.txt -inputFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_texturing/odm_textured_model.obj -outputFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/pmvs/recon0/models/option-0000.ply -outputPointCloudFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/odm_georeferenced_model.ply -logFile /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/odm_georeferencing_log.txt -georefFileOutputPath /home/mechanik/dev/odm_data/odm_boruszyn_kap/odm_georeferencing/odm_georeferencing_model_geo.txt
returned with code 256.

git rev-parse HEAD
2db0411b27dae745e29307dd8de1383caf152cb9

And there's no benchmark file. Maybe I can't find it in project folder.

[dakotabenjamin]

Check the new PR #264

[dakotabenjamin]

Merged #264

[merkato]

If we start ODM with --use-opensfm-pointcloud benchmark file looks like this:

Resizing runtime: 20.945608 seconds
Meshing runtime: 78.041546 seconds
Texturing runtime: 269.293121 seconds
Georeferencing runtime: 191.849991 seconds
Orthophoto runtime: 79.996433 seconds

I think we're missing OpenSfM sparse and dense reconstruction here.

git rev-parse HEAD
5b0cd53f2ecd7422ca5461cb14cc55eeb7c34fba