MicMac process fails to produce Orthophoto

[ChipwizBen]

How did you install OpenDroneMap? (Docker, natively, ...)?

Docker

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

N/A

What is the problem?

After processing a known-good GPS dataset, the Orthophoto failed to complete. Appropriate log lines:

No orthophoto found at odm_orthophoto/odm_orthophoto.tif: will skip MBTiles generation
Found suitable point cloud for PotreeConverter: odm_georeferencing/odm_georeferenced_model.ply

The 3D model completed successfully. What should be the expected behavior? If this is a feature request, please describe in detail the changes you think should be made to the code, citing files and lines where changes should be made, if possible.

Orthophoto should build correctly.

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

First dataset in here: https://community.opendronemap.org/t/poets-park-upper-hutt-new-zealand-x2/2042

Options: max-concurrency: 12, image-footprint: true, camera-cloud: true

[dronemapper-io]

We'll download the data and take a look. Is there any error messages or snippets from the log file that you can post? Thanks

[dronemapper-io]

It appears the images were post-processed after collection and some of the EXIF data is missing. You can try using the '--ccd-width' and '-ccd-height' options in order to force the camera ccd sensor size.

BEGIN Pre-compile
NEW CALIB TheKeyCalib__Foc-4700_Cam-FC220
WARN  !! , for camera FC220 cannot determine focale equiv-35mm
          add it in include/XML_User/DicoCamera.xml
For name = F7807128_5AB2_11E9_B590_5000EC033FF0.JPG
CANNOT GET ./Ori-FraserBasic/AutoCal_Foc--2147483647_Cam-FC220.xml
------------------------------------------------------------
|   Sorry, the following FATAL ERROR happened               
|                                                           
|    Cannot determine   calibration
|                                                           
------------------------------------------------------------

The raw images from the DJI sensor should process fine.

Or maybe try removing this file: F7807128_5AB2_11E9_B590_5000EC033FF0.JPG as its EXIF isn't valid

[dronemapper-io]

If you remove the file mentioned above the data set processes. (Shown below in Remote Expert - which uses customized MICMAC backend)

Low resolution gray scale ortho preview shown

[pierotofy]

Bad files tend to happen. We've been thinking about improving this by adding sanitization checks for a while in ODM. https://github.com/OpenDroneMap/ODM/issues/910 (just haven't had gotten around to making it, but this could be a common module).

[dronemapper-io]

Agreed.. a pre-processing QA QC module would benefit both engines!

[ChipwizBen]

Maybe I'm just extremely unlucky with the datasets that I put through MicMac, but every one of them works in ODM. For every dataset that has actually produced something in MicMac only the 3D model is ever produced - I've not had a single fully successful run that also produces an orthophoto.

Latest error:

[DEBUG]   running mm3d CenterBascule .*.JPG RadialStd RAWGNSS_N Ground_Init_RTL
BEGIN Pre-compile
NEW CALIB TheKeyCalib__Foc-8800_Cam-FC6310
WARN  !! , for camera FC6310 cannot determine focale equiv-35mm
add it in include/XML_User/DicoCamera.xml
For name = F0109054_9076_11E9_914B_49645C2213EA.JPG
CANNOT GET ./Ori-RadialStd/AutoCal_Foc-109054000_Cam-FC6310.xml
------------------------------------------------------------
|   Sorry, the following FATAL ERROR happened
|
|    Cannot determine   calibration
|
------------------------------------------------------------
-------------------------------------------------------------
|       (Elise's)  LOCATION :
|
| Error was detected
|          at line : 1341
|          of file : /var/www/micmac/src/uti_phgrm/Apero/cCalib.cpp
-------------------------------------------------------------
Bye  (press enter)

WebODM still believes that this is running also. I guess "Bye (press enter)" needs to be added to the console parser as a fail marker.

[dronemapper-io]

You should remove the image in question from the logfile alert. The images have been modified and pre-processed so some of the EXIF metadata is missing.

WARN  !! , for camera FC6310 cannot determine focale equiv-35mm
add it in include/XML_User/DicoCamera.xml
For name = F0109054_9076_11E9_914B_49645C2213EA.JPG

https://github.com/dronemapper-io/NodeMICMAC/issues/23#issuecomment-498660564

[ChipwizBen]

I can assure you that the images have not been preprocessed in any way and the issue was closed as solved before I had a chance to reply to that the first time around. They have been renamed to prevent overlaps with other datasets, that's all. The files are all stored on either ZFS or Ceph, with scrubbing turned on.

If they are missing data then that's because it either wasn't written correctly on the drone or was partially corrupted during copy from the SD card. Whichever reason it is, I don't have these issues with the exact same files in ODM.

[pierotofy]

Like it has been mentioned earlier, a sanitation module would improve things to capture corrupted data. But for the time being, just check your data before using NodeMicMac. ODM has been tested for much longer, so it's expected that ODM is a bit more tolerant with bad data.

Contributions welcome!

[dronemapper-io]

@ChipwizBen They should process for you if you remove the file mentioned or use the -ccd-width/height params to enforce the sensor size so the software can properly compute focal points/etc. ODM and MM are definitely different softwares so it is expected that certain datasets might work in one and not the other. If you still have troubles, please post the logfile and the params/options you are running the job with. Thanks

[ChipwizBen]

I removed the file mentioned. The Ortho is still not produced. Log is 20MB so can't attach here. I've dropped it here: https://dev.aerosurvey.co.nz/console.txt

[dronemapper-io]

Thanks for the log. It looks like the balanced Ortho gets about 1/2 way thru processing.. is there a potential that the server doesn't have enough resources or ran out of disk space?

Radiom Max  401.703 451.586 556.3
KBOX = 655 On 1178
OUT [8964,16597][9961,17573]
IN [8864,16497][10061,17673]

Radiom Max  401.703 451.586 556.3
KBOX = 656 On 1178
OUT [9961,16597][10957,17573]
IN [9861,16497][11057,17673]

Radiom Max  401.703 451.586 556.3
KBOX
Done!

[ChipwizBen]

I don't think so. It's not as big as the other nodes but it's still 64GB, 24 core. Disk space is 500G, should be plenty for these little surveys.

I'll hook it up to some monitoring just to make sure, but I'm pretty sure that it's not resourcing that's the issue.

[dronemapper-io]

I've made some updates to the ortho creation/color balance. https://github.com/dronemapper-io/NodeMICMAC/commit/4070a6eb28b13a9bd35588f6fb92934322b2370c

I would wait for docker hub to finish building the new image and give it a try. Thanks

[ChipwizBen]

I upped the memory to be sure, but it's definitely not memory that it's running out of. In fact, most of it is cache anyway:

Storage is also < 40% in all partitions.

[dronemapper-io]

I've reverted one other MicMac commit -- you can test it once the docker image is done building. https://github.com/dronemapper-io/micmac/commit/b9475765c20f8297630f4a4f3319451dfe08bba4

I would also recommend testing your entire process with the DroneMapper 4th Ave. example data set as well. Thanks

[ChipwizBen]

4th Ave works OK, but 4 other inputs from 3 different drones still fail to produce the ortho using the latest pull (as of a couple of days ago). Drones were Mavic Pro, P4, and Inspire 2. The same datasets work OK in NodeODM.

I'm really keen to find the cause of this and start using ModeMICMAC; is there anything else that I can check?

[kikislater]

Did you adjust dji altitude in exif ?

[ChipwizBen]

No. The files were completely unmodified from drone to MicMac.

[dronemapper-io]

4th Ave works OK, but 4 other inputs from 3 different drones still fail to produce the ortho using the latest pull (as of a couple of days ago). Drones were Mavic Pro, P4, and Inspire 2. The same datasets work OK in NodeODM.

I'm really keen to find the cause of this and start using ModeMICMAC; is there anything else that I can check?

That would point to a data collection issue .. It is hard to determine the exact cause without processing the entire dataset or the 3 other datasets .. Sorry about the bad news .. some data will work properly in ODM and not MicMac given they are very different processing softwares/methods. Thanks

[ChipwizBen]

Two were from Pix4Dcapture, one was using DroneDeploy. I could send you the datasets if you are interested. Should I send you an email with a link or is there a better alternative?

[dronemapper-io]

Ok thanks for the additional info. If you would like to post a link to the smallest data set I will try and work out the issue. If you need to keep the data / link confidential please contact us via https://dronemapper.com

[ChipwizBen]

Here are two: Buildings, 400 photos, 4.8G, Inspire 2: https://dashboard.aerosurvey.co.nz/files/shared/BT-397.tar.gz Landscape, 156 photos, 1.6G, Mavic 2 Pro: https://dashboard.aerosurvey.co.nz/files/shared/OK-242.tar.gz

[ChipwizBen]

Any news?

[dronemapper-io]

The latest version of nodeMicMac works for me:

I made a small update to the underlying MicMac Porto Params and you can give it a try once the latest docker image is built.

This is running directly on an Ubuntu/PopOS 19.04 machine. NodeMicMac only.

[ChipwizBen]

Thanks for looking into this. I tried the other dataset (buildings) but it still has the same issue. Updated as of 2 days ago.

[kikislater]

Reopen if still an issue