ModuleNotFoundError: No module named 'packaging' when running with Singularity

How did you install ODM? (Docker, installer, natively, ...)?

Singularity singularity pull --disable-cache docker://opendronemap/odm:latest singularity pull --disable-cache docker://opendronemap/odm:2.8.6

Using singularity 3.9.6 Currently Loaded Modulefiles: 1) singularity/3.9.6(default)

What is the problem?

I was testing out using Singularity to run the container for jobs on a HPC cluster but for some reason when I try to execute a simple test job I get the error below:

[sserbin@node03 odm_processing_code]$ singularity run --bind $images_dir:/$output_dir/code/images --writable-tmpfs odm_latest.sif --orthophoto-png --dtm --orthophoto-kmz --project-path $output_dir
[INFO]    DTM is turned on, automatically turning on point cloud classification
[INFO]    Initializing ODM ? - Tue Jul 19 04:12:41  2022
[INFO]    ==============
[INFO]    3d_tiles: False
[INFO]    auto_boundary: False
[INFO]    auto_boundary_distance: 0
[INFO]    boundary: {}
[INFO]    build_overviews: False
[INFO]    camera_lens: auto
[INFO]    cameras: {}
[INFO]    cog: False
[INFO]    copy_to: None
[INFO]    crop: 3
[INFO]    debug: False
[INFO]    dem_decimation: 1
[INFO]    dem_euclidean_map: False
[INFO]    dem_gapfill_steps: 3
[INFO]    dem_resolution: 5
[INFO]    depthmap_resolution: 640
[INFO]    dsm: False
[INFO]    dtm: True
[INFO]    end_with: odm_postprocess
[INFO]    fast_orthophoto: False
[INFO]    feature_quality: high
[INFO]    feature_type: sift
[INFO]    force_gps: False
[INFO]    gcp: None
[INFO]    geo: None
[INFO]    gps_accuracy: 10
[INFO]    ignore_gsd: False
[INFO]    matcher_neighbors: 0
[INFO]    matcher_type: flann
[INFO]    max_concurrency: 36
[INFO]    merge: all
[INFO]    mesh_octree_depth: 11
[INFO]    mesh_size: 200000
[INFO]    min_num_features: 10000
[INFO]    name: code
[INFO]    no_gpu: False
[INFO]    optimize_disk_space: False
[INFO]    orthophoto_compression: DEFLATE
[INFO]    orthophoto_cutline: False
[INFO]    orthophoto_kmz: True
[INFO]    orthophoto_no_tiled: False
[INFO]    orthophoto_png: True
[INFO]    orthophoto_resolution: 5
[INFO]    pc_classify: True
[INFO]    pc_copc: False
[INFO]    pc_csv: False
[INFO]    pc_ept: False
[INFO]    pc_filter: 2.5
[INFO]    pc_geometric: False
[INFO]    pc_las: False
[INFO]    pc_quality: medium
[INFO]    pc_rectify: False
[INFO]    pc_sample: 0
[INFO]    pc_tile: False
[INFO]    primary_band: auto
[INFO]    project_path: /data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm
[INFO]    radiometric_calibration: none
[INFO]    rerun: None
[INFO]    rerun_all: False
[INFO]    rerun_from: None
[INFO]    resize_to: 2048
[INFO]    rolling_shutter: False
[INFO]    rolling_shutter_readout: 0
[INFO]    sfm_algorithm: incremental
[INFO]    skip_3dmodel: False
[INFO]    skip_band_alignment: False
[INFO]    skip_orthophoto: False
[INFO]    skip_report: False
[INFO]    sky_removal: False
[INFO]    sm_cluster: None
[INFO]    smrf_scalar: 1.25
[INFO]    smrf_slope: 0.15
[INFO]    smrf_threshold: 0.5
[INFO]    smrf_window: 18.0
[INFO]    split: 999999
[INFO]    split_image_groups: None
[INFO]    split_overlap: 150
[INFO]    texturing_data_term: gmi
[INFO]    texturing_keep_unseen_faces: False
[INFO]    texturing_outlier_removal_type: gauss_clamping
[INFO]    texturing_skip_global_seam_leveling: False
[INFO]    texturing_skip_local_seam_leveling: False
[INFO]    texturing_tone_mapping: none
[INFO]    tiles: False
[INFO]    time: False
[INFO]    use_3dmesh: False
[INFO]    use_exif: False
[INFO]    use_fixed_camera_params: False
[INFO]    use_hybrid_bundle_adjustment: False
[INFO]    verbose: False
[INFO]    ==============
[INFO]    Running dataset stage
[INFO]    Loading dataset from: /data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm/code/images
[INFO]    Loading 187 images
[INFO]    Wrote images database: /data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm/code/images.json
[INFO]    Found 187 usable images
[INFO]    Parsing SRS header: WGS84 UTM 3N
[INFO]    Finished dataset stage
[INFO]    Running split stage
[INFO]    Normal dataset, will process all at once.
[INFO]    Finished split stage
[INFO]    Running merge stage
[INFO]    Normal dataset, nothing to merge.
[INFO]    Finished merge stage
[INFO]    Running opensfm stage
[INFO]    Maximum photo dimensions: 4056px
[INFO]    Photo dimensions for feature extraction: 2028px
[INFO]    nvidia-smi not found in PATH, using CPU
[INFO]    Altitude data detected, enabling it for GPS alignment
[INFO]    ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 2028', 'feature_min_frames: 10000', 'processes: 36', 'matching_gps_neighbors: 0', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: SIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0']
[INFO]    Wrote reference_lla.json
[INFO]    running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm/code/opensfm"
Traceback (most recent call last):
  File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 9, in <module>
    from opensfm import commands
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/__init__.py", line 1, in <module>
    from . import (
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/compute_statistics.py", line 4, in <module>
    from opensfm.actions import compute_statistics
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/compute_statistics.py", line 5, in <module>
    from opensfm import stats
  File "/code/SuperBuild/install/bin/opensfm/opensfm/stats.py", line 13, in <module>
    import matplotlib as mpl
  File "/home/sserbin/.local/lib/python3.9/site-packages/matplotlib/__init__.py", line 105, in <module>
    from packaging.version import parse as parse_version
ModuleNotFoundError: No module named 'packaging'

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 94, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 328, in run
    self.process(self.args, outputs)
  File "/code/stages/run_opensfm.py", line 35, in process
    octx.feature_matching(self.rerun())
  File "/code/opendm/osfm.py", line 410, in feature_matching
    raise e
  File "/code/opendm/osfm.py", line 400, in feature_matching
    self.run('detect_features')
  File "/code/opendm/osfm.py", line 34, in run
    system.run('"%s" %s "%s"' %
  File "/code/opendm/system.py", line 106, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

I've even simplified more

singularity run --bind $images_dir:/$output_dir/code/images --writable-tmpfs odm_latest.sif --orthophoto-png --dtm --orthophoto-kmz --project-path $output_dir

Same error

Interestingly the same source containers mostly work in podman but fail for another reason late in the processing, but I dont yet know why that is. I think perhaps an issue with trying to create the DTM. The example above does seem to generally work with podman/docker.

Not really sure why in Singularity it would be telling me a package is missing.

What should be the expected behavior? If this is a feature request, please describe in detail the changes you think should be made to the code, citing files and lines where changes should be made, if possible.

I would expect the job to execute and generate an orthophoto.

How can we reproduce this? What steps did you do to trigger the problem? If this is an issue with processing a dataset, YOU MUST include a copy of your dataset uploaded on Google Drive or Dropbox (otherwise we cannot reproduce this).

images_dir=/data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/images
name=`basename $images_dir`
output_dir=/data2/RS_GIS_Data/UAS_Data/NGEEArctic/UAS_Flights/Seward_2021/Skydio2/Kougarok/20210816/Flight_2/odm
mkdir -p $output_dir

singularity run --bind $images_dir:/$output_dir/code/images \
--writable-tmpfs odm_latest.sif \
--orthophoto-png --dtm --orthophoto-kmz --feature-quality ultra --max-concurrency 16 \
--use-hybrid-bundle-adjustment --build-overviews --min-num-features 10000 \
--project-path $output_dir

I suspect I may not be invoking the run correctly. Any insight would be much appreciated. Thanks!

OpenDroneMap / ODM