"An orthophoto could not be generated" (WebODM fails to sync with restarted node-odm process)

[mbandy]

Hi!

It is fairly common when processing larger datasets (150+ images) for node-OpenDroneMap to crash and need to be restarted. When restarted, node-ODM picks up where it left off, but WebODM never reacquires the output stream. In these cases I have set the task status to "Completed" manually. When this is done, the 3D products are available through the web interface but the orthophoto is not and the following error is displayed:

An orthophoto could not be generated. To generate one, make sure GPS information is embedded in the EXIF tags of your images.

The odm_orthophoto and orthophoto tiles directories are present in

WebODM/app/media/project/[]/task/[]

... but I can't figure out how to get the web interface to recognize and display them. Any suggestions?

[pierotofy]

Hi @mbandy, thanks for reporting the problem!

Is there a odm_orthophoto.tif (not png) file in WebODM/app/media/project/[]/task/[]/assets/ ?

It's odd that node-odm keeps crashing, 150+ images are not a whole lot. Do you know what the cause of the crash is?

[mbandy]

Hi!

There is indeed an odm_orthophoto.tifin WebODM/app/media/project/[]/task/[]/assets/odm_orthophoto/.

I have noticed though that there is usually a second .tif in that directory, for example odm_orthophoto_4326.tif. I'm not sure where that one comes from, but I'm pretty sure ODM does not generate it. It's missing from the problem task I am reporting.

I'm not sure what the cause of the crash is. It always crashes during the opensfm step, usually with datasets approaching 250 photos. It is probably the well-known opensfm out of memory error. So I think it's an ODM problem, not a problem with node-ODM. However, WebODM does not recover gracefully when node-ODM is restarted and reruns the ODM process.

[pierotofy]

If you are using docker, you should try to increase the memory allocated to the docker machine.

As far as the orthophoto is concerned, the case you are describing should have never happened (in theory). If assets have been downloaded (in /assets/odm_orthophoto) it means that odm_orthophoto_4326.tif should have been generated also.

                            orthophoto_path = os.path.realpath(self.assets_path("odm_orthophoto", "odm_orthophoto.tif"))
                            if os.path.exists(orthophoto_path):
                                orthophoto = GDALRaster(orthophoto_path, write=True)

                                # We need to transform to 4326 before we can store it
                                # as an offdb raster field
                                orthophoto_4326_path = os.path.realpath(self.assets_path("odm_orthophoto", "odm_orthophoto_4326.tif"))
                                self.orthophoto = orthophoto.transform(4326, 'GTiff', orthophoto_4326_path)

                                logger.info("Imported orthophoto {} for {}".format(orthophoto_4326_path, self))

I can't explain why everything was extracted from the all.zip archive, but the orthophoto was not set in the database. It should have, or it should have thrown an error.

When you manually modified the status to be completed, was it because WebODM errored out, or kept displaying error messages and never finished the task?

Is the output odm_orthophoto.tif a valid GeoTIFF (can you import it in a software such as QGIS)?

[pierotofy]

It would help tremendously to have a copy of the WebODM console log in this case, to further narrow down the problem.

[mbandy]

I should apologize and explain. Much of this is my own fault, and I'm not sure there is a bug at all. More like a suggested improvement to WebODM's error handling. Here is what happened, as I probably should have reported it originally:

1. node-ODM crashed as a result of ODM crashing during opensfm (I'm not using the docker version - I have everything compiled and running native, but only 32 gigs of RAM).
2. When I restarted node-ODM it picked up where it left off and finished sucessfully. However, WebODM continued to show the process as "running".
3. In an attempt to fix this, I changed the task status in WebODM to "completed" in the "Administration" tab. Of course, the assets were not in the right place so were not available via WebODM.
4. I copied the relevant directories and files from node-ODM/data/ to WebODM/app/media/project/[]/task/[]/assets/. That worked except for the orthophoto. This is the situation I described in my original post.
5. After you posted your code snippet, I finally understood the process. I used gdalwarp to convert odm_orthophoto.tif to -t_srs 4326 and generated odm_orthophoto_4326.tif manually. I then entered the path to that file in the "Administration" tab. Everything now functions as expected.

My apologies for generating confusion. I think the only bug here is in WebODM's failure to gracefully manage a node-ODM crash, but I don't have any real insight into how to fix it.

[pierotofy]

Ahh that makes more sense :)

I will leave this open because WebODM should have reattached to the node-odm process once the process came back online.

[pierotofy]

I can't seem to be able to reproduce this error. If I start a node-odm process via webodm, kill the node-odm process, then restart the node-odm process, webodm successfully picks up the task where it left off.

@mbandy if you can find a way to reproduce the sync problem, please report it here. I will close this issue for now.

[mbandy]

Thanks, @pierotofy. I'll try to recreate the error and I'll report the results here if they are useful.