search

OpenDroneMap / ODM

A command line toolkit to generate maps, point clouds, 3D models and DEMs from drone, balloon or kite images. 📷
https://opendronemap.org
GNU Affero General Public License v3.0
4.92k stars 1.11k forks source link

Failing process during sift I believe #1409

Closed DarkSlaayer closed 2 years ago

DarkSlaayer commented 2 years ago

How did you install OpenDroneMap? (Docker, natively, ...)?

Docker

What's your browser and operating system? (Copy/paste the output of https://www.whatismybrowser.com/)

Chrome 97 on Windows 11 Does it on Windows 10 also

What is the problem?

128GB Swap Fails when its around 60gb usually

Stopping during sift

How can we reproduce this? (What steps did you do to trigger the problem? What parameters are you using for processing? If possible please include a copy of your dataset uploaded on Google Drive or Dropbox. Be detailed)

`[INFO] Initializing ODM 2.7.2 - Mon Jan 24 04:31:29 2022 [INFO] ============== [INFO] auto_boundary: True [INFO] boundary: {} [INFO] build_overviews: False [INFO] camera_lens: auto [INFO] cameras: {} [INFO] cog: True [INFO] copy_to: None [INFO] crop: 3 [INFO] debug: True [INFO] dem_decimation: 1 [INFO] dem_euclidean_map: False [INFO] dem_gapfill_steps: 3 [INFO] dem_resolution: 5 [INFO] depthmap_resolution: 640 [INFO] dsm: True [INFO] dtm: True [INFO] end_with: odm_postprocess [INFO] fast_orthophoto: False [INFO] feature_quality: ultra [INFO] feature_type: sift [INFO] force_gps: False [INFO] gcp: None [INFO] geo: None [INFO] gps_accuracy: 10 [INFO] ignore_gsd: True [INFO] matcher_neighbors: 36 [INFO] matcher_type: flann [INFO] max_concurrency: 16 [INFO] merge: all [INFO] mesh_octree_depth: 12 [INFO] mesh_size: 200000 [INFO] min_num_features: 18000 [INFO] name: df6b7d2b-1e44-4f46-bd98-d2524e8aa4fc [INFO] optimize_disk_space: False [INFO] orthophoto_compression: DEFLATE [INFO] orthophoto_cutline: False [INFO] orthophoto_kmz: False [INFO] orthophoto_no_tiled: False [INFO] orthophoto_png: False [INFO] orthophoto_resolution: 1.0 [INFO] pc_classify: True [INFO] pc_csv: False [INFO] pc_ept: True [INFO] pc_filter: 0.0 [INFO] pc_geometric: True [INFO] pc_las: False [INFO] pc_quality: ultra [INFO] pc_rectify: True [INFO] pc_sample: 0 [INFO] pc_tile: False [INFO] primary_band: auto [INFO] project_path: /var/www/data [INFO] radiometric_calibration: none [INFO] rerun: None [INFO] rerun_all: False [INFO] rerun_from: ['dataset', 'split', 'merge', 'opensfm', 'openmvs', 'odm_filterpoints', 'odm_meshing', 'mvs_texturing', 'odm_georeferencing', 'odm_dem', 'odm_orthophoto', 'odm_report', 'odm_postprocess'] [INFO] resize_to: -1 [INFO] sfm_algorithm: incremental [INFO] skip_3dmodel: False [INFO] skip_band_alignment: False [INFO] skip_orthophoto: True [INFO] skip_report: False [INFO] sm_cluster: None [INFO] smrf_scalar: 1.25 [INFO] smrf_slope: 0.15 [INFO] smrf_threshold: 0.5 [INFO] smrf_window: 18.0 [INFO] split: 999999 [INFO] split_image_groups: None [INFO] split_overlap: 150 [INFO] texturing_data_term: gmi [INFO] texturing_keep_unseen_faces: False [INFO] texturing_outlier_removal_type: gauss_clamping [INFO] texturing_skip_global_seam_leveling: False [INFO] texturing_skip_local_seam_leveling: False [INFO] texturing_tone_mapping: none [INFO] tiles: False [INFO] time: False [INFO] use_3dmesh: True [INFO] use_exif: False [INFO] use_fixed_camera_params: False [INFO] use_hybrid_bundle_adjustment: False [INFO] verbose: True [INFO] ============== [INFO] Running dataset stage [INFO] Loading dataset from: /var/www/data/df6b7d2b-1e44-4f46-bd98-d2524e8aa4fc/images [DEBUG] /var/www/data/df6b7d2b-1e44-4f46-bd98-d2524e8aa4fc/images [INFO] Loading 456 images I removed the images from here due to lat/long coords [INFO] Wrote images database: /var/www/data/df6b7d2b-1e44-4f46-bd98-d2524e8aa4fc/images.json [INFO] Found 456 usable images [INFO] Parsing SRS header: WGS84 UTM 14N [INFO] Finished dataset stage [INFO] Running split stage [INFO] Normal dataset, will process all at once. [INFO] Finished split stage [INFO] Running merge stage [INFO] Normal dataset, nothing to merge. [INFO] Finished merge stage [INFO] Running opensfm stage [INFO] Maximum photo dimensions: 4864px [INFO] Photo dimensions for feature extraction: 4864px [WARNING] Image size (4864x3648px) would not fit in GPU memory, falling back to CPU [INFO] Altitude data detected, enabling it for GPS alignment [INFO] ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 4864', 'feature_min_frames: 18000', 'processes: 16', 'matching_gps_neighbors: 36', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: SIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0'] [INFO] Wrote reference_lla.json [INFO] running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/df6b7d2b-1e44-4f46-bd98-d2524e8aa4fc/opensfm" 2022-01-24 04:31:34,873 INFO: Capping memory usage to ~ 3186.419921875 MB 2022-01-24 04:31:34,873 INFO: Expecting to process 47 images. 2022-01-24 04:31:34,878 INFO: Reading data for image DJI_0448.JPG (queue-size=0 2022-01-24 04:31:34,878 INFO: Reading data for image DJI_0142.JPG (queue-size=0 2022-01-24 04:31:34,878 INFO: Reading data for image DJI_0282.JPG (queue-size=0 2022-01-24 04:31:34,878 INFO: Reading data for image DJI_0274.JPG (queue-size=0 2022-01-24 04:31:35,188 INFO: Reading data for image DJI_0235.JPG (queue-size=1 2022-01-24 04:31:35,188 INFO: Extracting ROOT_SIFT features for image DJI_0448.JPG 2022-01-24 04:31:35,194 INFO: Reading data for image DJI_0350.JPG (queue-size=1 2022-01-24 04:31:35,194 INFO: Extracting ROOT_SIFT features for image DJI_0282.JPG 2022-01-24 04:31:35,195 INFO: Reading data for image DJI_0226.JPG (queue-size=1 2022-01-24 04:31:35,196 INFO: Extracting ROOT_SIFT features for image DJI_0142.JPG 2022-01-24 04:31:35,197 INFO: Reading data for image DJI_0032.JPG (queue-size=1 2022-01-24 04:31:35,198 INFO: Extracting ROOT_SIFT features for image DJI_0274.JPG 2022-01-24 04:31:35,211 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,212 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,213 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,233 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,694 INFO: Reading data for image DJI_0349.JPG (queue-size=1 2022-01-24 04:31:35,695 INFO: Extracting ROOT_SIFT features for image DJI_0235.JPG 2022-01-24 04:31:35,722 INFO: Reading data for image DJI_0222.JPG (queue-size=1 2022-01-24 04:31:35,723 INFO: Extracting ROOT_SIFT features for image DJI_0226.JPG 2022-01-24 04:31:35,725 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,730 INFO: Reading data for image DJI_0247.JPG (queue-size=1 2022-01-24 04:31:35,731 INFO: Extracting ROOT_SIFT features for image DJI_0350.JPG 2022-01-24 04:31:35,733 INFO: Reading data for image DJI_0230.JPG (queue-size=1 2022-01-24 04:31:35,743 INFO: Extracting ROOT_SIFT features for image DJI_0032.JPG 2022-01-24 04:31:35,756 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,780 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:35,817 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:36,286 INFO: Reading data for image DJI_0447.JPG (queue-size=1 2022-01-24 04:31:36,287 INFO: Extracting ROOT_SIFT features for image DJI_0349.JPG 2022-01-24 04:31:36,329 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:36,332 INFO: Reading data for image DJI_0046.JPG (queue-size=1 2022-01-24 04:31:36,335 INFO: Reading data for image DJI_0181.JPG (queue-size=1 2022-01-24 04:31:36,336 INFO: Extracting ROOT_SIFT features for image DJI_0230.JPG 2022-01-24 04:31:36,337 INFO: Extracting ROOT_SIFT features for image DJI_0222.JPG 2022-01-24 04:31:36,358 INFO: Reading data for image DJI_0183.JPG (queue-size=1 2022-01-24 04:31:36,361 INFO: Extracting ROOT_SIFT features for image DJI_0247.JPG 2022-01-24 04:31:36,427 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:36,434 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:36,456 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:31:38,635 INFO: Reading data for image DJI_0011.JPG (queue-size=2 2022-01-24 04:31:38,654 INFO: Reading data for image DJI_0083.JPG (queue-size=3 2022-01-24 04:31:38,655 INFO: Reading data for image DJI_0232.JPG (queue-size=4 2022-01-24 04:31:38,632 INFO: Reading data for image DJI_0276.JPG (queue-size=2 2022-01-24 04:31:46,175 INFO: Reading data for image DJI_0207.JPG (queue-size=5 2022-01-24 04:31:46,617 INFO: Reading data for image DJI_0245.JPG (queue-size=6 2022-01-24 04:31:46,828 INFO: Reading data for image DJI_0020.JPG (queue-size=7 2022-01-24 04:31:46,996 INFO: Reading data for image DJI_0209.JPG (queue-size=8 2022-01-24 04:31:47,484 INFO: Reading data for image DJI_0246.JPG (queue-size=9 2022-01-24 04:31:47,497 INFO: Reading data for image DJI_0378.JPG (queue-size=10 2022-01-24 04:31:47,975 INFO: Reading data for image DJI_0063.JPG (queue-size=11 2022-01-24 04:31:48,020 INFO: Reading data for image DJI_0005.JPG (queue-size=12 2022-01-24 04:31:48,041 INFO: Reading data for image DJI_0113.JPG (queue-size=13 2022-01-24 04:31:48,586 INFO: Reading data for image DJI_0159.JPG (queue-size=14 2022-01-24 04:31:48,602 INFO: Reading data for image DJI_0254.JPG (queue-size=15 2022-01-24 04:31:48,631 INFO: Reading data for image DJI_0013.JPG (queue-size=16 2022-01-24 04:31:49,187 INFO: Reading data for image DJI_0012.JPG (queue-size=17 2022-01-24 04:31:49,211 INFO: Reading data for image DJI_0307.JPG (queue-size=18 2022-01-24 04:31:49,228 INFO: Reading data for image DJI_0353.JPG (queue-size=19 2022-01-24 04:31:49,776 INFO: Reading data for image DJI_0058.JPG (queue-size=20 2022-01-24 04:31:49,806 INFO: Reading data for image DJI_0194.JPG (queue-size=21 2022-01-24 04:31:49,833 INFO: Reading data for image DJI_0095.JPG (queue-size=22 2022-01-24 04:31:49,841 INFO: Reading data for image DJI_0255.JPG (queue-size=23 2022-01-24 04:31:50,882 INFO: Reading data for image DJI_0228.JPG (queue-size=24 2022-01-24 04:31:50,882 INFO: Reading data for image DJI_0272.JPG (queue-size=25 2022-01-24 04:31:51,562 INFO: Reading data for image DJI_0126.JPG (queue-size=26 2022-01-24 04:31:51,562 INFO: Reading data for image DJI_0200.JPG (queue-size=27 2022-01-24 04:31:52,319 INFO: Reading data for image DJI_0450.JPG (queue-size=28 2022-01-24 04:31:52,377 INFO: Reading data for image DJI_0455.JPG (queue-size=29 2022-01-24 04:31:52,377 INFO: Reading data for image DJI_0238.JPG (queue-size=30 2022-01-24 04:31:52,608 INFO: Reading data for image DJI_0155.JPG (queue-size=31 2022-01-24 04:31:53,226 INFO: Reading data for image DJI_0401.JPG (queue-size=32 2022-01-24 04:31:53,269 INFO: Reading data for image DJI_0273.JPG (queue-size=33 2022-01-24 04:31:53,280 INFO: Reading data for image DJI_0156.JPG (queue-size=34 2022-01-24 04:31:53,293 INFO: Reading data for image DJI_0295.JPG (queue-size=35 2022-01-24 04:31:54,316 INFO: Reading data for image DJI_0069.JPG (queue-size=36 2022-01-24 04:31:54,427 INFO: Reading data for image DJI_0380.JPG (queue-size=37 2022-01-24 04:31:54,429 INFO: Reading data for image DJI_0322.JPG (queue-size=38 2022-01-24 04:31:54,596 INFO: Reading data for image DJI_0221.JPG (queue-size=39 2022-01-24 04:31:55,147 INFO: Reading data for image DJI_0162.JPG (queue-size=40 2022-01-24 04:31:55,324 INFO: Reading data for image DJI_0358.JPG (queue-size=42 2022-01-24 04:31:55,325 INFO: Reading data for image DJI_0296.JPG (queue-size=43 2022-01-24 04:31:55,325 INFO: Reading data for image DJI_0331.JPG (queue-size=43 2022-01-24 04:31:55,830 INFO: Reading data for image DJI_0354.JPG (queue-size=44 2022-01-24 04:31:55,861 INFO: Reading data for image DJI_0143.JPG (queue-size=45 2022-01-24 04:31:55,861 INFO: Reading data for image DJI_0004.JPG (queue-size=46 2022-01-24 04:31:55,863 INFO: Reading data for image DJI_0382.JPG (queue-size=47 Traceback (most recent call last): File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 25, in commands.command_runner( File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command_runner.py", line 37, in command_runner command.run(data, args) File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command.py", line 12, in run self.run_impl(data, args) File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/detect_features.py", line 11, in run_impl detect_features.run_dataset(dataset) File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 15, in run_dataset features_processing.run_features_processing(data, data.images(), False) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 65, in run_features_processing 2022-01-24 04:33:57,112 INFO: Extracting ROOT_SIFT features for image DJI_0046.JPG parallel_map(process, arguments, processes, 1) File "/code/SuperBuild/install/bin/opensfm/opensfm/context.py", line 56, in parallel_map 2022-01-24 04:33:57,117 INFO: Extracting ROOT_SIFT features for image DJI_0447.JPG res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 1061, in call 2022-01-24 04:33:57,117 INFO: Extracting ROOT_SIFT features for image DJI_0181.JPG 2022-01-24 04:33:57,110 INFO: Extracting ROOT_SIFT features for image DJI_0183.JPG self.retrieve() File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 940, in retrieve self._output.extend(job.get(timeout=self.timeout)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 595, in call return self.func(*args, *kwargs) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in call return [func(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in return [func(*args, **kwargs) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 117, in process read_images(queue, data, images, counter, expected, force) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 141, in read_images queue.put(args, block=True, timeout=full_queue_timeout) File "/usr/lib/python3.8/queue.py", line 147, in put raise Full queue.Full terminate called recursively terminate called without an active exception terminate called without an active exception terminate called recursively /code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12: 1814 Aborted "$PYTHON" "$DIR"/opensfm_main.py "$@"

===== Dumping Info for Geeks (developers need this to fix bugs) ===== Child returned 134 Traceback (most recent call last): File "/code/stages/odm_app.py", line 94, in execute self.first_stage.run() File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 327, in run self.process(self.args, outputs) File "/code/stages/run_opensfm.py", line 35, in process octx.feature_matching(self.rerun()) File "/code/opendm/osfm.py", line 321, in feature_matching self.run('detect_features') File "/code/opendm/osfm.py", line 34, in run system.run('"%s" %s "%s"' % File "/code/opendm/system.py", line 106, in run raise SubprocessException("Child returned {}".format(retcode), retcode) opendm.system.SubprocessException: Child returned 134

===== Done, human-readable information to follow... =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/`

Also I have this one I do not have the full log due to me restarting it

`2022-01-24 04:18:09,179 INFO: Extracting ROOT_SIFT features for image DJI_0004.JPG 2022-01-24 04:18:11,739 DEBUG: Computing sift with threshold 0.066 2022-01-24 04:20:07,638 INFO: Reading data for image DJI_0278.JPG (queue-size=47 Traceback (most recent call last): File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 25, in commands.command_runner( File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command_runner.py", line 37, in command_runner 2022-01-24 04:20:07,804 INFO: Extracting ROOT_SIFT features for image DJI_0254.JPG command.run(data, args) File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command.py", line 12, in run self.run_impl(data, args) File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/detect_features.py", line 11, in run_impl detect_features.run_dataset(dataset) File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 15, in run_dataset features_processing.run_features_processing(data, data.images(), False) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 65, in run_features_processing parallel_map(process, arguments, processes, 1) File "/code/SuperBuild/install/bin/opensfm/opensfm/context.py", line 56, in parallel_map res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 1061, in call self.retrieve() File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 940, in retrieve self._output.extend(job.get(timeout=self.timeout)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 595, in call return self.func(*args, *kwargs) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in call return [func(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in return [func(*args, **kwargs) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 117, in process read_images(queue, data, images, counter, expected, force) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 141, in read_images queue.put(args, block=True, timeout=full_queue_timeout) File "/usr/lib/python3.8/queue.py", line 147, in put raise Full queue.Full

===== Dumping Info for Geeks (developers need this to fix bugs) ===== Child returned 1 Traceback (most recent call last): File "/code/stages/odm_app.py", line 94, in execute self.first_stage.run() File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 327, in run self.process(self.args, outputs) File "/code/stages/run_opensfm.py", line 35, in process octx.feature_matching(self.rerun()) File "/code/opendm/osfm.py", line 321, in feature_matching self.run('detect_features') File "/code/opendm/osfm.py", line 34, in run system.run('"%s" %s "%s"' % File "/code/opendm/system.py", line 106, in run raise SubprocessException("Child returned {}".format(retcode), retcode) opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/`

pierotofy commented 2 years ago

If possible please include a copy of your dataset uploaded on Google Drive or Dropbox

DarkSlaayer commented 2 years ago

Heres another pretty much any larger dataset its happening to.

[INFO] Wrote images database: /var/www/data/f30077d2-6638-49ac-95a8-427f709ecc12/images.json [INFO] Found 606 usable images [INFO] Parsing SRS header: WGS84 UTM 14N [INFO] Finished dataset stage [INFO] Running split stage [INFO] Large dataset detected (606 photos) and split set at 20. Preparing split merge. [INFO] Maximum photo dimensions: 4864px [INFO] Photo dimensions for feature extraction: 4864px [WARNING] Image size (4864x3648px) would not fit in GPU memory, falling back to CPU [INFO] Altitude data detected, enabling it for GPS alignment [INFO] ['use_exif_size: no', 'flann_algorithm: KDTREE', 'feature_process_size: 4864', 'feature_min_frames: 30000', 'processes: 16', 'matching_gps_neighbors: 36', 'matching_gps_distance: 0', 'matching_graph_rounds: 50', 'optimize_camera_parameters: yes', 'reconstruction_algorithm: incremental', 'undistorted_image_format: tif', 'bundle_outlier_filtering_type: AUTO', 'sift_peak_threshold: 0.066', 'align_orientation_prior: vertical', 'triangulation_type: ROBUST', 'retriangulation_ratio: 2', 'matcher_type: FLANN', 'feature_type: SIFT', 'use_altitude_tag: yes', 'align_method: auto', 'local_bundle_radius: 0', 'submodels_relpath: ../submodels/opensfm', 'submodel_relpathtemplate: ../submodels/submodel%04d/opensfm', 'submodel_images_relpathtemplate: ../submodels/submodel%04d/images', 'submodel_size: 20', 'submodel_overlap: 150'] [INFO] Wrote reference_lla.json [INFO] running "/code/SuperBuild/install/bin/opensfm/bin/opensfm" detect_features "/var/www/data/f30077d2-6638-49ac-95a8-427f709ecc12/opensfm" 2022-01-27 04:29:32,631 INFO: Expecting to process 65 images. 2022-01-27 04:29:32,637 INFO: Reading data for image DJI_0669.JPG (queue-size=0 2022-01-27 04:29:32,637 INFO: Reading data for image DJI_0054-edit.JPG (queue-size=0 2022-01-27 04:29:32,637 INFO: Reading data for image DJI_0255.JPG (queue-size=0 2022-01-27 04:29:32,637 INFO: Reading data for image DJI_0417.JPG (queue-size=0 2022-01-27 04:29:33,076 INFO: Reading data for image DJI_0071-edit.JPG (queue-size=1 2022-01-27 04:29:33,077 INFO: Extracting ROOT_SIFT features for image DJI_0255.JPG 2022-01-27 04:29:33,083 INFO: Reading data for image DJI_0325.JPG (queue-size=1 2022-01-27 04:29:33,084 INFO: Extracting ROOT_SIFT features for image DJI_0054-edit.JPG 2022-01-27 04:29:33,091 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,096 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,103 INFO: Reading data for image DJI_0678.JPG (queue-size=1 2022-01-27 04:29:33,103 INFO: Extracting ROOT_SIFT features for image DJI_0417.JPG 2022-01-27 04:29:33,110 INFO: Reading data for image DJI_0448.JPG (queue-size=1 2022-01-27 04:29:33,115 INFO: Extracting ROOT_SIFT features for image DJI_0669.JPG 2022-01-27 04:29:33,127 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,163 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,629 INFO: Reading data for image DJI_0430.JPG (queue-size=1 2022-01-27 04:29:33,630 INFO: Extracting ROOT_SIFT features for image DJI_0325.JPG 2022-01-27 04:29:33,643 INFO: Reading data for image DJI_0450.JPG (queue-size=1 2022-01-27 04:29:33,644 INFO: Extracting ROOT_SIFT features for image DJI_0071-edit.JPG 2022-01-27 04:29:33,657 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,693 INFO: Reading data for image DJI_0590.JPG (queue-size=1 2022-01-27 04:29:33,693 INFO: Extracting ROOT_SIFT features for image DJI_0678.JPG 2022-01-27 04:29:33,694 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,756 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:33,790 INFO: Reading data for image DJI_0543.JPG (queue-size=1 2022-01-27 04:29:33,791 INFO: Extracting ROOT_SIFT features for image DJI_0448.JPG 2022-01-27 04:29:33,851 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:34,440 INFO: Reading data for image DJI_0494.JPG (queue-size=1 2022-01-27 04:29:34,443 INFO: Extracting ROOT_SIFT features for image DJI_0450.JPG 2022-01-27 04:29:34,441 INFO: Reading data for image DJI_0041-edit.JPG (queue-size=2 2022-01-27 04:29:34,444 INFO: Extracting ROOT_SIFT features for image DJI_0430.JPG 2022-01-27 04:29:34,507 INFO: Reading data for image DJI_0007-edit.JPG (queue-size=1 2022-01-27 04:29:34,509 INFO: Extracting ROOT_SIFT features for image DJI_0590.JPG 2022-01-27 04:29:34,517 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:34,517 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:34,606 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:34,665 INFO: Reading data for image DJI_0015-edit.JPG (queue-size=1 2022-01-27 04:29:34,667 INFO: Extracting ROOT_SIFT features for image DJI_0543.JPG 2022-01-27 04:29:34,776 DEBUG: Computing sift with threshold 0.066 2022-01-27 04:29:35,485 INFO: Reading data for image DJI_0419.JPG (queue-size=1 2022-01-27 04:29:35,658 INFO: Reading data for image DJI_0504.JPG (queue-size=2 2022-01-27 04:29:35,717 INFO: Reading data for image DJI_0588.JPG (queue-size=3 2022-01-27 04:29:36,004 INFO: Reading data for image DJI_0349.JPG (queue-size=4 2022-01-27 04:29:37,283 INFO: Reading data for image DJI_0463.JPG (queue-size=5 2022-01-27 04:29:37,310 INFO: Reading data for image DJI_0401.JPG (queue-size=6 2022-01-27 04:29:37,340 INFO: Reading data for image DJI_0248.JPG (queue-size=7 2022-01-27 04:29:37,658 INFO: Reading data for image DJI_0680.JPG (queue-size=8 2022-01-27 04:29:39,569 INFO: Reading data for image DJI_0340.JPG (queue-size=9 2022-01-27 04:29:39,831 INFO: Reading data for image DJI_0291.JPG (queue-size=10 2022-01-27 04:29:39,840 INFO: Reading data for image DJI_0582.JPG (queue-size=11 2022-01-27 04:29:40,079 INFO: Reading data for image DJI_0538.JPG (queue-size=12 2022-01-27 04:29:43,680 INFO: Reading data for image DJI_0580.JPG (queue-size=13 2022-01-27 04:29:49,389 INFO: Reading data for image DJI_0069.JPG (queue-size=14 2022-01-27 04:29:49,392 INFO: Reading data for image DJI_0465.JPG (queue-size=15 2022-01-27 04:29:49,396 INFO: Reading data for image DJI_0054.JPG (queue-size=16 2022-01-27 04:29:50,110 INFO: Reading data for image DJI_0435.JPG (queue-size=17 2022-01-27 04:29:50,423 INFO: Reading data for image DJI_0521.JPG (queue-size=18 2022-01-27 04:29:50,442 INFO: Reading data for image DJI_0331.JPG (queue-size=19 2022-01-27 04:29:50,502 INFO: Reading data for image DJI_0641.JPG (queue-size=20 2022-01-27 04:29:50,695 INFO: Reading data for image DJI_0631.JPG (queue-size=21 2022-01-27 04:29:51,636 INFO: Reading data for image DJI_0447.JPG (queue-size=22 2022-01-27 04:29:51,952 INFO: Reading data for image DJI_0493.JPG (queue-size=23 2022-01-27 04:29:52,374 INFO: Reading data for image DJI_0025-edit.JPG (queue-size=24 2022-01-27 04:29:52,374 INFO: Reading data for image DJI_0428.JPG (queue-size=25 2022-01-27 04:29:53,363 INFO: Reading data for image DJI_0361.JPG (queue-size=26 2022-01-27 04:29:53,363 INFO: Reading data for image DJI_0020.JPG (queue-size=27 2022-01-27 04:29:53,467 INFO: Reading data for image DJI_0037-edit.JPG (queue-size=28 2022-01-27 04:29:53,467 INFO: Reading data for image DJI_0700.JPG (queue-size=29 2022-01-27 04:29:54,515 INFO: Reading data for image DJI_0546.JPG (queue-size=30 2022-01-27 04:29:54,521 INFO: Reading data for image DJI_0552.JPG (queue-size=31 2022-01-27 04:29:54,525 INFO: Reading data for image DJI_0442.JPG (queue-size=32 2022-01-27 04:29:54,595 INFO: Reading data for image DJI_0258.JPG (queue-size=33 2022-01-27 04:29:55,433 INFO: Reading data for image DJI_0246.JPG (queue-size=34 2022-01-27 04:29:55,433 INFO: Reading data for image DJI_0377.JPG (queue-size=35 2022-01-27 04:29:55,433 INFO: Reading data for image DJI_0630.JPG (queue-size=36 2022-01-27 04:29:55,617 INFO: Reading data for image DJI_0292.JPG (queue-size=37 2022-01-27 04:29:56,161 INFO: Reading data for image DJI_0048.JPG (queue-size=38 2022-01-27 04:29:56,165 INFO: Reading data for image DJI_0029.JPG (queue-size=39 2022-01-27 04:29:56,184 INFO: Reading data for image DJI_0005.JPG (queue-size=40 2022-01-27 04:29:56,186 INFO: Reading data for image DJI_0458.JPG (queue-size=41 2022-01-27 04:29:56,739 INFO: Reading data for image DJI_0656.JPG (queue-size=42 2022-01-27 04:29:56,765 INFO: Reading data for image DJI_0530.JPG (queue-size=43 2022-01-27 04:29:56,811 INFO: Reading data for image DJI_0013.JPG (queue-size=44 2022-01-27 04:29:56,827 INFO: Reading data for image DJI_0241.JPG (queue-size=45 2022-01-27 04:29:57,735 INFO: Reading data for image DJI_0014.JPG (queue-size=46 2022-01-27 04:29:57,735 INFO: Reading data for image DJI_0376.JPG (queue-size=47 2022-01-27 04:29:57,740 INFO: Reading data for image DJI_0064.JPG (queue-size=48 2022-01-27 04:29:57,745 INFO: Reading data for image DJI_0003-edit.JPG (queue-size=49 2022-01-27 04:29:58,880 INFO: Reading data for image DJI_0343.JPG (queue-size=50 2022-01-27 04:29:58,881 INFO: Reading data for image DJI_0681.JPG (queue-size=51 2022-01-27 04:29:58,886 INFO: Reading data for image DJI_0649.JPG (queue-size=52 2022-01-27 04:29:58,886 INFO: Reading data for image DJI_0617.JPG (queue-size=53 2022-01-27 04:29:59,641 INFO: Reading data for image DJI_0027.JPG (queue-size=54 2022-01-27 04:29:59,644 INFO: Reading data for image DJI_0345.JPG (queue-size=55 2022-01-27 04:29:59,651 INFO: Reading data for image DJI_0307.JPG (queue-size=56 2022-01-27 04:29:59,655 INFO: Reading data for image DJI_0591.JPG (queue-size=57 2022-01-27 04:30:00,620 INFO: Reading data for image DJI_0426.JPG (queue-size=58 2022-01-27 04:30:00,629 INFO: Reading data for image DJI_0263.JPG (queue-size=59 2022-01-27 04:30:00,632 INFO: Reading data for image DJI_0513.JPG (queue-size=60 2022-01-27 04:30:00,661 INFO: Reading data for image DJI_0049-edit.JPG (queue-size=61 2022-01-27 04:30:01,324 INFO: Reading data for image DJI_0059-edit.JPG (queue-size=62 2022-01-27 04:30:01,324 INFO: Reading data for image DJI_0516.JPG (queue-size=63 2022-01-27 04:30:01,325 INFO: Reading data for image DJI_0037.JPG (queue-size=64 2022-01-27 04:30:01,326 INFO: Reading data for image DJI_0544.JPG (queue-size=65 Traceback (most recent call last): File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 25, in 2022-01-27 04:32:02,423 INFO: Extracting ROOT_SIFT features for image DJI_0041-edit.JPG commands.command_runner( File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command_runner.py", line 37, in command_runner 2022-01-27 04:32:02,432 INFO: Extracting ROOT_SIFT features for image DJI_0007-edit.JPG 2022-01-27 04:32:02,438 INFO: Extracting ROOT_SIFT features for image DJI_0494.JPG command.run(data, args) File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command.py", line 12, in run self.run_impl(data, args) File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/detect_features.py", line 11, in run_impl detect_features.run_dataset(dataset) File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 15, in run_dataset features_processing.run_features_processing(data, data.images(), False) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 65, in run_features_processing parallel_map(process, arguments, processes, 1) File "/code/SuperBuild/install/bin/opensfm/opensfm/context.py", line 56, in parallel_map res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 1061, in call self.retrieve() File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 940, in retrieve self._output.extend(job.get(timeout=self.timeout)) File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get raise self._value File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, kwds)) File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 595, in call return self.func(*args, *kwargs) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in call return [func(args, kwargs) File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in return [func(*args, **kwargs) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 117, in process read_images(queue, data, images, counter, expected, force) File "/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py", line 141, in read_images queue.put(args, block=True, timeout=full_queue_timeout) File "/usr/lib/python3.8/queue.py", line 147, in put raise Full queue.Full terminate called recursively terminate called recursively terminate called recursively /code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12: 81 Aborted "$PYTHON" "$DIR"/opensfm_main.py "$@"

===== Dumping Info for Geeks (developers need this to fix bugs) ===== Child returned 134 Traceback (most recent call last): File "/code/stages/odm_app.py", line 94, in execute self.first_stage.run() File "/code/opendm/types.py", line 346, in run self.next_stage.run(outputs) File "/code/opendm/types.py", line 327, in run self.process(self.args, outputs) File "/code/stages/splitmerge.py", line 60, in process octx.feature_matching(self.rerun()) File "/code/opendm/osfm.py", line 321, in feature_matching self.run('detect_features') File "/code/opendm/osfm.py", line 34, in run system.run('"%s" %s "%s"' % File "/code/opendm/system.py", line 106, in run raise SubprocessException("Child returned {}".format(retcode), retcode) opendm.system.SubprocessException: Child returned 134

===== Done, human-readable information to follow... =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

holyCowMp3 commented 2 years ago

Similar behavior in my case too :( Used nodeodm:gpu with latest updates.

DarkSlaayer commented 2 years ago

I increased full_queue_timeout = 120 to 120000 in features_processing.pay and it seems to be working. Obviously it’s not a correct fix but maybe we need to get a speed dataset on the average users page file and give it a little headroom?

Now I don’t have a full grasp on the inner workings of odm and I never had this issue until recently even tried native windows odm and same issue. But I think part of the problem is when using page filing/swap file and it times out before it finishes due to the slow nature of page filing.