Program quit after 4 days causing MemoryError

[yiboLi]

Hi there, I run the OpenDroneMap for about 4 days now, and it quit with the following information:

_Adding 2C185.JPG to the reconstruction Ceres Solver Report: Iterations: 2, Initial cost: 1.903508e+04, Final cost: 1.902635e+04, Termination: CONVERGENCE setup/run/teardown 10.643091917/138.229104996/3.02575802803 Remove 0 outliers Ceres Solver Report: Iterations: 0, Initial cost: 1.902635e+04, Final cost: 1.902635e+04, Termination: CONVERGENCE setup/run/teardown 10.6954760551/67.3890230656/3.02318787575 Reprojection Error: 0.00020471557183 Painting the reconstruction from 666 cameras Done. Reconstruction 0 : 666 images , 541437 points 1 partial reconstructions in total. Traceback (most recent call last): File "/odm_app/OpenDroneMap/src/OpenSfM/bin/mesh", line 32, in data.save_reconstruction(reconstructions, filename='reconstruction.meshed.json') File "/odm_app/OpenDroneMap/src/OpenSfM/opensfm/dataset.py", line 297, in save_reconstruction fout.write(io.json_dumps(reconstruction)) File "/odm_app/OpenDroneMap/src/OpenSfM/opensfm/io.py", line 209, in json_dumps return json.dumps(data, indent=indent, ensure_ascii=False).encode(codec) File "/usr/lib/python2.7/json/init.py", line 250, in dumps _sort_keys=sort_keys, **kw).encode(obj) File "/usr/lib/python2.7/json/encoder.py", line 210, in encode return ''.join(chunks) MemoryError b quitting cause: PYTHONPATH=/odm_app/OpenDroneMap/lib/python2.7/dist-packages "/odm_app/OpenDroneMap/src/OpenSfM/bin/run_all" opensfm returned with code 256.__

My questions are:

1. Configurations of my computer are: Processor: Intel(R) Core(TM) i7-4770K CPU @ 3.50 GHz RAM: 32.0 GB System type: 64-bit Operating System, x64-based processor And I have 666 image to process with. Maybe it quits because of the RAM? It is not big enough?
2. If the hardware configure of my computer is good enough, maybe it is because of the virtual machine? I allocate 28GB memory to VM, but I don't think this is the reason because while the program is running, it doesn't occupy too much memory of my computer.
3. Maybe there are some other problems of the code? I can't figure out right now (All the test files in odm_data work just fine). Does someone else has similar issue?

[dakotabenjamin]

1. That should be able to run the program just fine
2. Again that is definitely enough memory
3. If all the files in odm_data work fine, then I suspect that the large size of your dataset might be the problem.

Possible solutions:

1. create a subset area from your photos and run that. I would try 100 photos first, then 300, then 500.
2. resize the photos to something more manageable using the --resize-to x argument. I wouldn't resize them to smaller than 1200
3. Are your photos geotagged? Do they have lat-lon in the exif? OpenSfM has a built-in prematching step that will reduce the number of photos to match using a k-nearest neighbors algorithm, but it relies on geotagged images. There are scripts for geotagging images from gpx files, and others as well.

How do we fix this in the code? We need to look at the memory costs in OpenSfM and look into ways of optimizing that. How does OpenSfM scale with image size? number of images? key points?

We also need to implement the georeferencing step before OpenSfM so that the pre-matching has something to work with when photos don't have lat-lon in the exif, but the use provides a GCP file.

[yiboLi]

Thank you, that is a very good answer! My images don't have lat-lon in the exif file. I will try to resize them first and run smaller set of images. I will also update my results (or hopefully not more issues) in the following days.

[smathermather]

@yiboLi -- do you have any GPS info from the flight that you can correlate with the images?

[yiboLi]

@smathermather No, I don't have GPS info...

Today it generates another bug: _---- Initial: 0 secs ---- Total pass fail0 fail1 refinepatch: 3628765 318388 2630693 679684 998072 Total pass fail0 fail1 refinepatch: 100 8.774 72.4955 18.7304 27.5045 Expanding patches... ---- EXPANSION: 1715 secs ---- Total pass fail0 fail1 refinepatch: 7324629 6621973 310741 391915 7013888 Total pass fail0 fail1 refinepatch: 100 90.4069 4.24241 5.35065 95.7576 FilterOutside mainbody: Gain (ave/var): 2.0497 1.50949 6810373 -> 6527966 (95.8533%) 0 secs Filter Exact: **Killed b

quitting cause: "/odmapp/OpenDroneMap/bin/pmvs2" pmvs/ option-0000 returned with code 35072.

Does anyone has any clue what is going on?

[dakotabenjamin]

Was this on the original 666 images, or did you try my suggestions? What command did you run?

[dakotabenjamin]

As suggested in #28, try changing https://github.com/OpenDroneMap/OpenDroneMap/blob/gh-pages/run.py#L796 to be:

    run("\"" + BIN_PATH + "/genOption\" pmvs/ " + str(args.pmvs_level) + " " + str(args.pmvs_csize) + " " + str(args.pmvs_threshold) + " " + str(args.pmvs_wsize) + " " + str(args.pmvs_minImageNum) + " " + str(CORES - 1))

Also in #28, they mention that RAM may be an issue, and the number of cores should be similar to how many GB of RAM you have. I don't think we ever figured it out though.

[yiboLi]

@dakotabenjamin It was on original 666 images (I haven't resized them yet). I disabled openSFM because I already have the output results of openSFM from the last time, so the program started from CMVS. I tried to change the line to CORES - 1 as you said, but same error occurred. Should I try resize them to 1200X1200 (now it is 2400X1600) and run OpenDroneMap from the beginning? Another question: Maybe the storage of the disk is also a issue? Now it is the default value (virtual size: 40GB, actual size: 1.42GB) Should I assign more to the storage?

[dakotabenjamin]

That might fix it.

What command were you running exactly?

[yiboLi]

I just follow the test steps to run run.py. What about the VM storage?is this gonna be a problem?

[dakotabenjamin]

1.42 GB for 666 images is definitely too small