Error when using GPU Docker version

[John2397]

I recently started working with the docker ODM. I've successfully created the 3D model of this dataset by executing

docker run -it --rm \
    -v "$(pwd)/images:/code/images" \
    -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" \
    -v "$(pwd)/odm_texturing:/code/odm_texturing" \
    opendronemap/odm

Then, I've used the GPU Docker Version and tried the same dataset. Now the command becomes:

docker run -it --rm    \
 -v "$(pwd)/images:/code/images"   \
 -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" \
 -v "$(pwd)/odm_texturing:/code/odm_texturing"  \ 
  --gpus all opendronemap/odm:gpu

Running the above produces the following error:

...
2022-01-04 15:04:57,609 INFO: Reading data for image dji_0042.jpg (queue-size=17
/code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12:    41 Segmentation fault      (core dumped) "$PYTHON" "$DIR"/opensfm_main.py "$@"
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 139
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 94, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 346, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 346, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 346, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 327, in run
    self.process(self.args, outputs)
  File "/code/stages/run_opensfm.py", line 35, in process
    octx.feature_matching(self.rerun())
  File "/code/opendm/osfm.py", line 321, in feature_matching
    self.run('detect_features')
  File "/code/opendm/osfm.py", line 34, in run
    system.run('"%s" %s "%s"' %
  File "/code/opendm/system.py", line 106, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 139

===== Done, human-readable information to follow... =====
[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

After trying several datasets I've noticed that the above error rises only when I use the GPU docker version AND the dataset images contain their coordinates in EXIF. If they don't contain their coordinates in EXIF, the GPU version works fine.

My question is, is this some kind of a bug or is it something obvious that I'm not seeing?

[rumenmitrev]

i am seeing the same behaviour. I have been build my own docker image with latest repo.

EDIT1: The issue is with commit ab6986395eacdb2e9b3e128184644eba4d77c80b When i rollback numpy version to numpy==1.21.1 and bump rasterio==1.2.3, image is workin as expected

[holyCowMp3]

Same from my side! Similar behavior of GPU ODM version

[pierotofy]

It indeed seems to be an issue related to numpy.

So strange.

I'm running some tests to see if downgrading is an option; the reason we upgraded to 1.22.0 was to fix another issue. (https://community.opendronemap.org/t/updated-webodm-on-linux-now-wont-process-dataset/9966/8)

[pierotofy]

This should be fixed with #1402 (version 2.7.1). Thanks for the findings @rumenmitrev :+1: I upgraded rasterio and downgraded numpy as you mentioned and it seems to work. Will need to wait for the docker images to be updated and then run some more tests to confirm.

[rumenmitrev]

i have made tests with numpy==1.21.5(for GPU image only) and it seems everything is OK.

[holyCowMp3]

This should be fixed with #1402 (version 2.7.1). Thanks for the findings @rumenmitrev :+1: I upgraded rasterio and downgraded numpy as you mentioned and it seems to work. Will need to wait for the docker images to be updated and then run some more tests to confirm.

Confirmed 👍