search

OpenDroneMap / ODM

A command line toolkit to generate maps, point clouds, 3D models and DEMs from drone, balloon or kite images. 📷
https://opendronemap.org
GNU Affero General Public License v3.0
4.71k stars 1.08k forks source link

MemoryError after reconstruction #296

Closed GoshyK closed 6 years ago

GoshyK commented 8 years ago

HI,

I have use personal pictures (with GPS coordinates) for some tests and ODM works very well on my Ubuntu 15.10 OS with small dataset (I have succeeded with 160 pics). This is a very good soft, thanks to all developpers!!!

But when I try with large amount of pictures it fails with this message:

...
Adding DJI_2681.JPG to the reconstruction
Ceres Solver Report: Iterations: 1, Initial cost: 1.294364e+04, Final cost: 1.294146e+04, Termination: CONVERGENCE
setup/run/teardown 17.4406809807/65.4204010963/5.78068780899
Remove 0 outliers
Ceres Solver Report: Iterations: 0, Initial cost: 1.294146e+04, Final cost: 1.294146e+04, Termination: CONVERGENCE
setup/run/teardown 16.949919939/58.837638855/5.74048399925
Reprojection Error: 0.0001521000342
Painting the reconstruction from 922 cameras
Done.
Reconstruction 0 : 922 images , 1509172 points
1 partial reconstructions in total.
Traceback (most recent call last):
  File "/home/user/OpenDroneMap/src/OpenSfM/bin/mesh", line 32, in <module>
    data.save_reconstruction(reconstructions, filename='reconstruction.meshed.json')
  File "/home/user/OpenDroneMap/src/OpenSfM/opensfm/dataset.py", line 297, in save_reconstruction
    fout.write(io.json_dumps(reconstruction))
  File "/home/user/OpenDroneMap/src/OpenSfM/opensfm/io.py", line 209, in json_dumps
    return json.dumps(data, indent=indent, ensure_ascii=False).encode(codec)
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 210, in encode
    return ''.join(chunks)
MemoryError
b

quitting cause: 
    PYTHONPATH=/home/user/OpenDroneMap/lib/python2.7/dist-packages "/home/user/OpenDroneMap/src/OpenSfM/bin/run_all" opensfm
returned with code 256.

At this point the matches are found and the reconstruction has been done too but the result file reconstruction-with-image-size-2400-results is not created yet.

My questions are:

Many thanks,

dakotabenjamin commented 8 years ago
  1. how many images?
  2. Can you see if pre-matching step happens in opensfm? it would be near the beginning of that step
  3. some parameters to try: 
--min-num-features <integer>
                      Minimum number of features to extract per image. More
                      features leads to better results but slower execution.
                      Default: 6000

My guess is that there are too many images. The most I've ever tried is ~700, which took many hours on a 32-core 44GB ram machine.

GoshyK commented 8 years ago

Hi @Dakotabenjamin,

  1. I tried to compute a set of 2600 images, and another with 726images, both ended with the same error while another set of 165 images run well.
  2. How should I do that? I think that the matching proceeded well, the problem occurs later at the beginning of the reconstruction step.
  3. I am running a 922 images batch with --min-num-features 2000 I'll post the resu lt as soon as possible.
GoshyK commented 8 years ago

This is the end of the terminal computation info...

So, I think it just is too much images, and the parameters isn’t solving the problem.

3190 dog done
done
adding seeds 
(290,596)(293,230)(142,632)(11,997)(350,586)(322,16)(17,1054)(2,155)(110,1243)(209,1084)(69,1571)(361,915)(164,1315)(116,134)(143,277)(153,826)(115,1528)(75,1053)(59,779)(281,2320)(363,672)(166,1196)(57,1225)(329,321)(133,914)(27,613)(273,39)(275,858)(352,734)(192,368)(104,812)(114,235)(280,5)(288,90)(252,530)(326,425)(227,645)(96,1167)(341,700)(201,1353)(213,1445)(328,154)(79,650)(226,1162)(176,713)(163,138)(66,1102)(16,267)(182,202)(202,30)(259,1048)(0,616)(105,215)(171,155)(165,693)(149,647)(49,577)(106,1221)(237,126)(141,87)(354,4)(270,1337)(310,0)(168,106)(311,0)(33,250)(196,210)(318,1236)(212,6)(82,55)(25,704)(8,659)(43,1400)(23,2)(122,52)(345,113)(249,1033)(358,563)(287,140)(126,128)(120,232)(31,813)(1,3)(53,1162)(19,429)(44,1006)(264,188)(247,1406)(332,617)(238,475)(193,0)(5,166)(225,26)(4,608)(218,325)(180,1)(265,103)(155,415)(251,70)(131,915)(339,253)(338,203)(255,1438)(223,47)(48,0)(113,202)(127,4)(179,394)(67,0)(61,91)(263,15)(355,0)(157,1164)(190,146)(158,83)(232,672)(331,645)(151,7)(92,395)(337,159)(74,235)(159,427)(297,19)(200,77)(125,2)(301,39)(243,552)(187,25)(50,19)(309,30)(283,27)(124,246)(244,61)(368,163)(305,402)(170,44)(181,0)(188,930)(12,1)(46,6)(154,60)(118,110)(205,82)(334,577)(147,4)(99,101)(32,0)(210,0)(240,46)(156,1)(233,52)(161,92)(221,31)(266,137)(135,8)(178,120)(172,178)(211,0)(236,160)(145,26)(276,17)(365,1021)(14,12)(28,60)(22,7)(37,65)(185,0)(250,1)(36,2)(29,10)(204,494)(195,2)(268,14)(34,27)(144,0)(346,0)(242,15)(235,17)(356,2)(64,23)(121,1)(335,16)(94,14)(278,9)(207,4)(272,41)(123,10)(150,5)(208,1)(70,0)(294,5)(217,51)(97,42)(224,0)(9,0)(60,0)(175,13)(6,5)(186,38)(41,372)(128,29)(137,0)(289,0)(87,7)(129,5)(320,353)(26,4)(245,33)(279,38)(138,0)(162,19)(72,17)(359,0)(277,0)(93,0)(85,329)(241,36)(83,5)(160,0)(45,5)(198,1)(327,4)(73,3)(167,0)(291,0)(76,2)(261,14)(103,4)(84,189)(304,41)(367,20)(222,0)(119,7)(197,1)(174,2)(112,0)(206,0)(262,0)(184,0)(47,2)(111,0)(256,8)(78,318)(347,36)(215,77)(308,0)(130,1)(148,3)(340,5)(292,0)(35,0)(100,19)(214,5)(353,0)(58,33)(306,12)(91,1)(312,30)(7,0)(134,0)(315,0)(271,0)(253,0)(364,73)(95,0)(323,49)(319,4)(183,0)(13,0)(136,0)(370,7)(239,0)(313,0)(307,0)(330,39)(228,0)(258,2)(219,120)(257,0)(146,0)(321,4)(102,11)(234,0)(230,125)(317,17)(68,1)(51,5)(349,0)(88,0)(42,62)(348,1)(109,0)(189,0)(333,0)(366,1)(117,1)(89,5)(267,0)(343,13)(107,21)(246,0)(274,0)(56,68)(15,0)(3,1)(284,0)(269,0)(152,2)(360,0)(177,1)(81,0)(132,0)(299,0)(65,3)(21,0)(173,26)(24,1)(20,0)(54,0)(139,13)(260,0)(285,0)(86,0)(357,0)(248,0)(39,24)(362,0)(286,50)(55,11)(314,5)(303,10)(203,1)(295,5)(342,0)(10,0)(369,0)(229,4)(71,9)(344,70)(62,2)(38,1)(77,2)(298,0)(316,0)(169,0)(216,7)(30,0)(220,12)(63,0)(101,0)(90,0)(40,1)(52,0)(302,0)(336,1)(18,0)(194,0)(254,0)(108,0)(199,0)(80,0)(231,1)(140,0)(351,1)(296,0)(325,10)(300,0)(282,13)(324,5)(98,33)(191,34)done
---- Initial: 0 secs ----
Total pass fail0 fail1 refinepatch: 3060431 95747 2531752 432932 528679
Total pass fail0 fail1 refinepatch: 100 3.12855 82.7253 14.1461 17.2747
Expanding patches...
---- EXPANSION: 2036 secs ----
Total pass fail0 fail1 refinepatch: 10817774 9887013 187701 743060 10630073
Total pass fail0 fail1 refinepatch: 100 91.396 1.73512 6.86888 98.2649
FilterOutside
mainbody: 
Gain (ave/var): 0.746079 0.361089
9952232 -> 9940986 (99.887%)    0 secs
Filter Exact: ***********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
9940986 -> 9922453 (99.8136%)   0 secs
FilterNeighbor: 9922453 -> 9899826 (99.772%)    0 secs
FilterGroups:   989
9899826 -> 9841045 (99.4062%)   0 secs
STATUS: 212689 0 10945650 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 0 0 0 0 0 
0 0 0 0 0 
Expanding patches...
---- EXPANSION: 108 secs ----
Total pass fail0 fail1 refinepatch: 280262 150566 37257 92439 243005
Total pass fail0 fail1 refinepatch: 100 53.7233 13.2936 32.9831 86.7064
FilterOutside
mainbody: 
Gain (ave/var): 0.969485 0.41237
9991611 -> 9991556 (99.9995%)   0 secs
Filter Exact: Killed
b

quitting cause: 
    "/home/folder/OpenDroneMap/bin/pmvs2" pmvs/ option-0000
returned with code 35072.
dakotabenjamin commented 8 years ago

This seems to be a common problem. I think right now you can try to reduce image size using --resize-to to maybe 1500. You can also increase the number of cmvs clusters by decreasing the max images in each cluster: --cmvs-maxImages 200.

We have discussed possible options to replacing PMVS, as you can see here it is a limiting factor in a fully functional product. At the very least, it should be able to handle memory problems better.

alexhagiopol commented 8 years ago

I'm also having this exact issue with larger datasets and more demanding reconstruction settings.

dakotabenjamin commented 8 years ago

I'm currently working through and documenting the effects of various pmvs parameters on large datasets. I'll post them in a new issue and link everyone who has been having issues to corroborate my findings. Hopefully that will help.

In the meantime, if you go to the beginning of the pmvs step in your log, you will see a summary of options:

option-0000
--------------------------------------------------
--- Summary of specified options ---
# of timages: 296 (enumeration)
# of oimages: 0 (enumeration)
level: 1  csize: 1
threshold: 0.7  wsize: 7
minImageNum: 3  CPU: 32
useVisData: 1  sequence: -1
--------------------------------------------------

Is useVisData: 1 or is it 0?

GoshyK commented 8 years ago

I've tried the upper settings --resize-to 1500 & --cmvs-maxImages 200. This make the code run completely, without memroy error, but the first result I had today is not satisfying. The textured mesh is not good, and the orthophoto is not complete. I've run this with an extract of 900 images out of my 2600 original images. I will try with the full dataset as soon as possible!

dakotabenjamin commented 8 years ago

That is bittersweet news and makes it more pressing for us to finish the mvs-texturing-addition branch. Are you getting something like the image on the left below?

If so, the texturing improvements are in the pipeline and I think will be to your satisfaction.

GoshyK commented 8 years ago

This is what I get. There is a lack in light corrections and in the projection/position.

comparaison odm du 2016-06-08 10-15-26

dakotabenjamin commented 8 years ago

what did you use to get the right image?

it looks exactly like I expected as far as lighting/blending.

Now, for the projection- Have you looked at the ortho in map software? I think the georeferenced TIFF should be in UTM coords, so make sure you are comparing apples to apples. It will look warped if comparing planar and angular coordinate systems.

GoshyK commented 8 years ago

I tried Pix4D with 2600 images (if I remember well, or was it with the selection of 922 images?)

The left part of the above picture is as the ortho shows in QGIS in MTM 10 coords, but there is no difference with shotwell or imagemagick viewers.

Also, this is the Meshlab snapshot of my odm_textured_model_geo.obj ...weird!

snapshot00
dakotabenjamin commented 8 years ago

Something odd is definitely happening with the georeferencing.

dakotabenjamin commented 8 years ago

Wait, is the snapshot just upside down? i.e are you looking at the underside of the mesh?

GoshyK commented 8 years ago

Haha, no it isn't, I've turned it 360 upside down... same black color!

Date: Wed, 15 Jun 2016 07:54:04 -0700 From: notifications@github.com To: OpenDroneMap@noreply.github.com CC: goshiki_kohaku@hotmail.com; author@noreply.github.com Subject: Re: [OpenDroneMap/OpenDroneMap] MemoryError after reconstruction (#296)

Wait, is the snapshot just upside down? i.e are you looking at the underside of the mesh?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

dataludus commented 8 years ago

Hi all, I'm posting here because I get: [ERROR] quitting cause: /home/winner/Downloads/OpenDroneMap/SuperBuild/install/bin/pmvs2 /home/winner/Downloads/OpenDroneMap/odm_data_seneca/pmvs/recon0/ option-0000 returned with code 35072. after running the process with seneca data. The entire output is in attachment, but basically I see that there are 167 images and only 90 are used:

Kept: 1 4 7 8 9 10 11 13 14 15 18 19 21 22 23 24 26 28 29 30 32 33 34 36 38 41 42 43 44 45 46 47 50 51 53 54 55 56 58 59 60 61 62 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 82 83 85 86 88 89 90 91 92 93 95 96 97 98 99 100 101 102 103 104 105 106 108 109 110 111 112 113 114 115

Removed: 0 2 3 5 6 12 16 17 20 25 27 31 35 37 39 40 48 49 52 57 63 81 84 87 94 107 116

At the end, I see "DoG running. . . XXXX dog done" 90 times but right after comes: done Killed [ERROR]

Do you have any thoughs? log_process.txt

Fi156 commented 8 years ago

I think, that the termination has nothing todo with the removed photos.

"Missing photos" issue: append something like --matcher-neighbors 20 --min-num-features 1200

"Killed" issue: PMVS eats your RAM. With this image number you will need at least 10-16gb... Just try it with --resize-to 1000

dataludus commented 8 years ago

Thank you @Fi156, my problem was the RAM, with the resize parameter the process was completed.

On the other side, I don't know if the "Missing Photos" is actually an issue or not. Does this software uses the 100% of the photos always? For example, now I run the "caliterra" data, which has 77 images and I see that it only uses 31 (see attachment as well):

[INFO]    Running ODM OpenSfM Cell - Finished
[INFO]    Running ODM CMVS Cell
[DEBUG]   Writing CMVS vis in: /home/winner/Downloads/OpenDroneMap/odm_data_caliterra/pmvs/recon0/bundle.rd.out
[DEBUG]   running /home/winner/Downloads/OpenDroneMap/SuperBuild/install/bin/cmvs /home/winner/Downloads/OpenDroneMap/odm_data_caliterra/pmvs/recon0/ 500 1
Reading bundle...77 cameras -- 15152 points in bundle file
***********
77 cameras -- 15152 points
Reading images: *****************************************************************************
Set widths/heights...done   0 secs
done    0 secs
slimNeighborsSetLinks...done    0 secs
mergeSFM...***********resetPoints...done
Rep counts: 15152 -> 299    0 secs
setScoreThresholds...done   0 secs
sRemoveImages... ***********
Kept: 11 25 26 27 28 35 36 38 40 41 42 45 47 48 49 51 52 53 54 58 59 60 63 64 65 66 67 69 70 73 74 

Removed: 0 1 2 3 4 5 6 7 8 9 10 12 13 14 15 16 17 18 19 20 21 22 23 24 29 30 31 32 33 34 37 39 43 44 46 50 55 56 57 61 62 68 71 72 75 76 
sRemoveImages: 77 -> 31 0 secs
slimNeighborsSetLinks...done    0 secs

Cluster sizes: 
31 
Adding images: 
0 
Image nums: 77 -> 31 -> 31
Divide: 
done    0 secs
22.5161 images in vis on the average

[INFO]    Running ODM CMVS Cell - Finished
[INFO]    Running OMD PMVS Cell
[DEBUG]   Creating dense pointcloud in: /home/winner/Downloads/OpenDroneMap/odm_data_caliterra/pmvs/recon0/models/option-0000.ply
[DEBUG]   running /home/winner/Downloads/OpenDroneMap/SuperBuild/install/bin/genOption /home/winner/Downloads/OpenDroneMap/odm_data_caliterra/pmvs/recon0/ 1 2 0.7 7 3 1
[DEBUG]   running /home/winner/Downloads/OpenDroneMap/SuperBuild/install/bin/pmvs2 /home/winner/Downloads/OpenDroneMap/odm_data_caliterra/pmvs/recon0/ option-0000

/home/winner/Downloads/OpenDroneMap/SuperBuild/install/bin/pmvs2
/home/winner/Downloads/OpenDroneMap/odm_data_caliterra/pmvs/recon0/
option-0000
--------------------------------------------------
--- Summary of specified options ---
# of timages: 31 (enumeration)
# of oimages: 0 (enumeration)
level: 1  csize: 2
threshold: 0.7  wsize: 7
minImageNum: 3  CPU: 1
useVisData: 1  sequence: -1
--------------------------------------------------
Reading images: *******************************

log_process_caliterra.txt

dakotabenjamin commented 8 years ago

You're right, PMVS does not use all the photos. That is because the reconstructions may be partial sets- if your pmvs directory has more than one subdirectory (recon0, recon1, etc) then there's a chance not all the photos are being used to reconstruct. The reason this happens is usually because of too-low overlap, but there are other possibilities as well.

dataludus commented 8 years ago

Thank for your reply @dakotabenjamin. By the way, is the wiki the oficial "user manual"? I would like also to know if we need to modify the "camera_models.json" file under the opensfm/ folder in order to use another camera. In this file we have only a "v2 canon powershot elph" (this is a really old one! --> link). If this is true, how can we obtain all the parameters?

dakotabenjamin commented 6 years ago

This issue is outdated and will be closed. Please open a new thread in http://community.opendronemap.org and reference this issue if you would like to continue the conversation.