/bin/sh: 1: /root/OpenDroneMap/SuperBuild/build/pdal/bin/pdal: not found Error

[aganjoo]

While processing the data for a test mission, the process exit with the following error message in the WebODM console and status Process Failed exit code 1 Any suggestions or guidance is greatly appreciated

[DEBUG] running /root/OpenDroneMap/SuperBuild/build/pdal/bin/pdal pipeline -i /root/node-OpenDroneMap/data/b0c57ae9-e64b-4347-9408-bde3f5de861f/odm_georeferencing/pipeline.xml --readers.ply.filename=/root/node-OpenDroneMap/data/b0c57ae9-e64b-4347-9408-bde3f5de861f/odm_georeferencing/odm_georeferenced_model.ply --writers.las.filename=/root/node-OpenDroneMap/data/b0c57ae9-e64b-4347-9408-bde3f5de861f/odm_georeferencing/odm_georeferenced_model.ply.las /bin/sh: 1: /root/OpenDroneMap/SuperBuild/build/pdal/bin/pdal: not found Traceback (most recent call last): File "/root/OpenDroneMap/run.py", line 55, in plasm.execute(niter=1) File "/root/OpenDroneMap/scripts/odm_georeferencing.py", line 150, in process tree.odm_georeferencing_pdal) File "/root/OpenDroneMap/opendm/types.py", line 208, in convert_to_las '--writers.las.filename={f_out}'.format(**kwargs)) File "/root/OpenDroneMap/opendm/system.py", line 28, in run raise Exception("Child returned {}".format(retcode)) Exception: Child returned 127

[pierotofy]

It seems something went wrong in your build of OpenDroneMap. Could you share more details about your setup (how did you build/install OpenDroneMap and WebODM, did you use docker, does OpenDroneMap run from the command line or does the error happen only when you use WebODM)?

[aganjoo]

Thanks Pier.. We installed it natively on ubuntu 16.04. We are not using docker.

We cloned the latest ODM repo and ran configure.sh and the whole setup process executed without any errors. The error is coming whether we runa project directly on command line or using webodm. On checking it seems that there the folder structure /root/OpenDroneMap/SuperBuild/build/pdal/bin exists but the bin folder is empty

On a side note, we installed WedODM natively as well and discovered that requirements.txt lists Django version 0.80 which throws error during pip install. It needs to be version 0.90. After making that change webodm installs and runs correctly

[pierotofy]

For some reason it's possible that the pdal project did not build; when you do a:

$ ls /code/SuperBuild/build/pdal/bin                        
pdal  pdal-config

You should have pdal and pdal-config in the bin directory.

Perhaps to a clean build and copy/paste the resulting log as an attachment to this issue?

Would you be able to share what error you were getting during pip install? Also note that the current version in use is Django==1.10, not 0.80. Are you referring to django-common-helpers==0.8.0 instead?

[aganjoo]

Sorry, yes I meant django-common-helpers==0.8.0. We had to change that to 0.90. I will see if I saved the previous pip install

I will also run a new ODM build and paste the log shortly

[aganjoo]

Build ODM again and noticed this error in the log file

FindGDAL.cmake: gdal-config not found. Please set it manually: GDALCONFIG=GDAL CMake Error at cmake/gdal.cmake:13 (message): GDAL support is required Call Stack (most recent call first): CMakeLists.txt:221 (include)

make[2]: [pdal/stamp/pdal-configure] Error 1 make[1]: [CMakeFiles/pdal.dir/all] Error 2 make[1]: Waiting for unfinished jobs.... ^Cmake[5]: [CMakeFiles/csfm.dir/csfm.cc.o] Interrupt make[4]: [CMakeFiles/csfm.dir/all] Interrupt make[3]: [all] Interrupt make[2]: [opensfm/stamp/opensfm-build] Interrupt make[1]: [CMakeFiles/opensfm.dir/all] Interrupt

Then I ran the command to see if gdal is installed root@workspace:~/OpenDroneMap# dpkg -l | grep gdal ii gdal-bin 2.1.2+dfsg-2~xenial3 ii libgdal1i 1.11.3+dfsg-3build2 ii libgdal20 2.1.2+dfsg-2~xenial3 ii python-gdal 2.1.2+dfsg-2~xenial3 ii python3-gdal 2.1.2+dfsg-2~xenial3

Any thoughts

[aganjoo]

Also, here is the pip install log for WebODM Collecting django-common-helpers==0.8.0 (from -r requirements.txt (line 5)) Using cached django-common-helpers-0.8.0.tar.gz Complete output from command python setup.py egg_info: Traceback (most recent call last): File "", line 1, in File "/tmp/pip-build-s0b8rlpc/django-common-helpers/setup.py", line 16, in long_description = f.read().strip() File "/root/WebODM/venv/lib/python3.5/encodings/ascii.py", line 26, in decode return codecs.ascii_decode(input, self.errors)[0] UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 5737: ordinal not in range(128)

----------------------------------------

Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-s0b8rlpc/django-common-helpers/

[pierotofy]

Here's the dpkg from the docker image:

root@49a95409271d:/var/www# dpkg -l | grep gdal
ii  gdal-bin                               1.10.1+dfsg-5ubuntu1                    amd64        Geospatial Data Abstraction Library - Utility programs
ii  libgdal-dev                            1.10.1+dfsg-5ubuntu1                    amd64        Geospatial Data Abstraction Library - Development files
ii  libgdal1h                              1.10.1+dfsg-5ubuntu1                    amd64        Geospatial Data Abstraction Library
ii  python-gdal                            1.10.1+dfsg-5ubuntu1                    amd64        Python bindings to the Geospatial Data Abstraction Library

PDAL is looking for GDAL 1.9.0, or a compatible version such as 1.10.1. You have 2.1.2, which probably is not compatible. Downgrade GDAL to 1.10.1 and try to run the build process again.

Per /code/SuperBuild/src/pdal/cmake/gdal.cmake

find_package(GDAL 1.9.0)

Thanks for the error report; I've actually ran some tests, and django-common-helpers is not currently being used anymore in WebODM, so I'm going to remove it.

[aganjoo]

I reinstalled and ODM is working fine with GDAL 1.9 but now WebODM does not run and complains that it needs GDAL 2.1 or higher when start.sh script is run

[pierotofy]

WebODM requires 2.1, because of a feature needed by off-db raster support which is available only in gdal 2.

I think available options include:

• Installing both 1.9 and 2.1, then starting WebODM via: export PATH=/path/to/gdal2.1/:$PATH ./start.sh
• Installing OpenDroneMap and WebODM on separate machines
• Use docker to run WebODM.

The third option is probably the easiest (if you know docker).

[pierotofy]

I'll close this for now, unless there are further questions.

[aganjoo]

Thanks for all the replies and apologies for the delayed response. We were able to get everything running by using multiple machines and splitting up ODM and WebODM. We were able to run a couple of sample datasets and they worked great but the last data set we ran resulted in WebODM webserver dying after image upload with the message below (venv) root@workspace:~/WebODM#

INFO Acquiring lock: Task ID: 1 INFO Processing... Task ID: 1 ./start.sh: line 58: 9732 Killed python manage.py runserver 0.0.0.0:8000

[1]+ Done chmod +x start.sh && ./start.sh

We tried to restart the services but it keeps dying with the same message. No matter what we try webodm wont start now

Any thoughts on what might be happening

[pierotofy]

What happens if you simply run:

python manage.py runserver 0.0.0.0:8000

?

[aganjoo]

It dies, see below

(venv) root@workspace:~/WebODM# python manage.py runserver 0.0.0.0:8000 Performing system checks...

INFO Starting background scheduler... System check identified no issues (0 silenced). February 06, 2017 - 21:08:47 Django version 1.10, using settings 'webodm.settings' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. INFO Acquiring lock: Task ID: 1 INFO Processing... Task ID: 1 Killed (venv) root@workspace:~/WebODM#

[pierotofy]

Let's see if we can retrieve a core dump for more information:

Try to run:

ulimit -c unlimited
python manage.py runserver 0.0.0.0:8000

Do you get a "core dumped" message after that? If so, look for a file called core. Please attach that file to this discussion.

[aganjoo]

No core dump was generated

(venv) root@workspace:~/WebODM# ulimit -c unlimited (venv) root@workspace:~/WebODM# python manage.py runserver 0.0.0.0:8000 Performing system checks...

INFO Starting background scheduler... System check identified no issues (0 silenced). February 07, 2017 - 08:55:22 Django version 1.10, using settings 'webodm.settings' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. INFO Acquiring lock: Task ID: 1 INFO Processing... Task ID: 1 Killed

[pierotofy]

What if you try to use pdb?

python -m pdb manage.py runserver 0.0.0.0:8000

Once the process is killed, type w to print a stack trace.

[aganjoo]

(venv) root@workspace:~/WebODM# python -m pdb manage.py runserver 0.0.0.0:8000

/root/WebODM/manage.py(2)() -> import os (Pdb) (Pdb) w /usr/lib/python3.5/bdb.py(431)run() -> exec(cmd, globals, locals)

(1)() /root/WebODM/manage.py(2)() -> import os (Pdb)

[pierotofy]

Ah, you need to type r before w. r will run the program, w will print the stack trace. https://docs.python.org/2/library/pdb.html

[aganjoo]

(venv) root@workspace:~/WebODM# python -m pdb manage.py runserver 0.0.0.0:8000

/root/WebODM/manage.py(2)() -> import os (Pdb) r Performing system checks...

INFO Starting background scheduler... System check identified no issues (0 silenced). February 07, 2017 - 09:39:59 Django version 1.10, using settings 'webodm.settings' Starting development server at http://0.0.0.0:8000/ Quit the server with CONTROL-C. wINFO Acquiring lock: Task ID: 1 INFO Processing... Task ID: 1

Killed

[pierotofy]

Mm, no luck there. Could you try to run:

python manage.py test

?

[aganjoo]

No Joy see below...If it helps when I drop all the webodm realated tables and resetup webodm it starts fine but when processing the data set the problem shows up again as soon as it start s processing the uploaded images

(venv) root@workspace:~/WebODM# python manage.py test Creating test database for alias 'default'... Got an error creating the test database: permission denied to create database

Type 'yes' if you would like to try deleting the test database 'test_anra-odm', or 'no' to cancel: yes Destroying old test database for alias 'default'... Got an error recreating the test database: database "test_anra-odm" does not exist

[pierotofy]

If the problem only starts to happen with a certain dataset, but works fine with others, would you be able to share (even privately) the dataset so that I could try to reproduce the issue?

[pierotofy]

The fact that the test suite fails in this manner is suspicious however. If I was to start troubleshooting this, I would probably start by trying to get the test suite to run without errors.

[aganjoo]

I will send you a link to download the dataset offline. I was able to process it using ODM directly without issues

[aganjoo]

can you send me your email offline so I can send you a link or suggest a way to get you the dataset

[pierotofy]

Use this form: https://www.masseranolabs.com/contact/

[pierotofy]

How much RAM does your WebODM and node-ODM machine have?

[aganjoo]

4GB

[pierotofy]

I was able to process the dataset on my laptop with 24GB of ram, default settings.

My guess is that you don't have enough memory available for WebODM to process the images, and the process is getting killed by the OS.

Perhaps try to manually connect to Postgres and remove the task manually so that you can get up and running again. The tasks are stored in the app_task table.

[smathermather]

You might also consider going into advanced options and set use-opensfm-pointcloud: true for a better quality output (you'll still need more RAM than 4GB, as docker isn't going to be able to take advantage of all of that.

[aganjoo]

Thanks Gents.. I am going to try these steps tonight. Out in the field today