search

OpenDroneMap / NodeMICMAC

A Lightweight REST API to Access MICMAC Photogrammetry and SFM Engine.
https://opendronemap.org/nodemicmac/
GNU General Public License v3.0
79 stars 21 forks source link

Only one CPU is used #40

Closed u5000 closed 2 years ago

u5000 commented 4 years ago

Hello and thank you for all your work.

I have a machine with 32 CPUs (each 1 core) and i can't figure out how to make NodeMICMAC to use more than one CPU. Is there a special command for that? I found the parameter to tell it how many cores it should use, but now how many CPUs.

I use docker+webodm+nodemicmac

thank you again.

dronemapper-io commented 4 years ago

Hello: You should use this parameter:

  --max-concurrency <integer>
                        The maximum number of cores to use in processing.
                        Default: 4

Certain processes in the pipeline are single threaded won't use more than 1 thread/CPU. For example, the alignment / calibration step and the orthomosaic creation step. Thanks

u5000 commented 4 years ago

yea was doing this (with number 32). But only ever one CPU was used (looked at the CPU rate very often).

One project took about 20 hours. Were with the normal webodm node it would only take 1-2 hours.

Im using a virtual server with 32 CPUs for clarification.

dronemapper-io commented 4 years ago

Can you post the parameters you are using?

On Wed, Nov 6, 2019, 9:01 AM u5000 notifications@github.com wrote:

yea was doing this (with number 32). But only ever one CPU was used (looked at the CPU rate very often).

One project took about 20 hours. Where with the normal webodm node it would only take 1-2 hours.

Im using a virtual server with 32 CPUs for clarification.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dronemapper-io/NodeMICMAC/issues/40?email_source=notifications&email_token=ALIJ6KRRQGFMRCR34763LPLQSLS67A5CNFSM4JJWYEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDHBL3Q#issuecomment-550376942, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIJ6KS2KREVRD7GKRKZT63QSLS67ANCNFSM4JJWYEKA .

u5000 commented 4 years ago

I did several tests:

Using the webgui of WebODM:

Using the webgui of nodemicmac directly:

All of those attempts never raised the CPU usage above one CPU.

As i said, my setup has 32 CPUs, which are utilized by the odm_node but not on micmac.

dronemapper-io commented 4 years ago

Thanks for the details. When using NodeMICMAC what are you resizing the images to? Can you share (privately if you want) a few example images and log export? If you watch via terminal using the 'top' command, the feature extraction and DEM construction should be multi-threaded. Thanks

On Thu, Nov 7, 2019, 5:20 AM u5000 notifications@github.com wrote:

I did several tests:

Using the webgui of WebODM:

-

ccd-width: 13.2, max-concurrency: 32, multi-scale: true, ccd-height: 8.8, image-footprint: true, camera-cloud: true

ccd-width: 13.2, max-concurrency: 32, matcher-distance: 50, ccd-height: 8.8

Default settings (at least 4 CPUs should have been used here)

Using the webgui of nodemicmac directly:

  • ccd-width: 13.2, max-concurrency: 32, multi-scale: true, ccd-height: 8.8, image-footprint: true, camera-cloud: true

All of those attempts never raised the CPU usage above one CPU.

As i said, my setup has 32 CPUs, which are utilized by the odm_node but not on micmac.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dronemapper-io/NodeMICMAC/issues/40?email_source=notifications&email_token=ALIJ6KWMKDX3NUJXOW7OZNLQSQB2RA5CNFSM4JJWYEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDMHMVA#issuecomment-551056980, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIJ6KXIDGPARNZPY255TFTQSQB2RANCNFSM4JJWYEKA .

u5000 commented 4 years ago

Sorry for the late response. Yea i did watched the top command. I just updated everything again and it might be faster now, i need a few days to observe. Anyway i think there is a lot of potential for further parallelisation, since nodeodm is a lot faster (and uses more CPU time) in comparison.

dronemapper-io commented 4 years ago

Thanks. Keep us posted. Sure, there are improvements that can be made. I would suggest looking into making Porto multi-threaded and/or making Apero multi-threaded -- no small task! Submit a PR and contributions are always welcome.

On Mon, Nov 11, 2019 at 7:54 AM u5000 notifications@github.com wrote:

Sorry for the late response. Yea i did watched the top command. I just updated everything again and it might be faster now, i need a few days to observe. Anyway i think there is a lot of potential for further parallelisation, since nodeodm is a lot faster (and uses more CPU time) in comparison.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dronemapper-io/NodeMICMAC/issues/40?email_source=notifications&email_token=ALIJ6KR3SIJPFV2SE5OJC5TQTFWYZA5CNFSM4JJWYEKKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDXCBSI#issuecomment-552476873, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALIJ6KRQQYG7MWP3RHEYTXDQTFWYZANCNFSM4JJWYEKA .

kikislater commented 2 years ago

Latest version of micmac is now in this repository, I closed this one