SIFT_GPU fails with very large images (>11000 pixels wide)

[pierotofy]

2021-10-29 15:45:22,869 INFO: Finished reading images
2021-10-29 15:45:22,871 INFO: Extracting ROOT_SIFT_GPU features for image image111.jpg
Traceback (most recent call last):
  File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 15, in <module>
    commands.command_runner(
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command_runner.py", line 38, in command_runner
    command.run(data, args)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command.py", line 12, in run
    self.run_impl(data, args)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/detect_features.py", line 11, in run_impl
    detect_features.run_dataset(dataset)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 68, in run_dataset
    parallel_map(process, arguments, processes, 1)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/context.py", line 52, in parallel_map
    res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 1061, in __call__
    self.retrieve()
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 940, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in __call__
    return [func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in <listcomp>
    return [func(*args, **kwargs)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 137, in process
    run_detection(queue)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 171, in run_detection
    detect(image, image_array, segmentation_array, instances_array, data)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 231, in detect
    unmasked, keypoints = features.extract_features(
  File "/code/SuperBuild/install/bin/opensfm/opensfm/features.py", line 570, in extract_features
    points, desc, keypoints = extract_features_sift_gpu(image, config, features_count)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/features.py", line 354, in extract_features_sift_gpu
    keypoints = get_sift_gpu().detect_image(image)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/sift_gpu.py", line 63, in detect_image
    return self.gpu_sift(image)
  File "/usr/local/lib/python3.8/dist-packages/silx/opencl/sift/plan.py", line 517, in keypoints
    kp, descriptor = self._one_octave(octave)
  File "/usr/local/lib/python3.8/dist-packages/silx/opencl/sift/plan.py", line 593, in _one_octave
    self._gaussian_convolution(self.cl_mem["scale_%i" % scale], self.cl_mem["scale_%i" % (scale + 1)], sigma, octave)
  File "/usr/local/lib/python3.8/dist-packages/silx/opencl/sift/plan.py", line 564, in _gaussian_convolution
    k2 = self.kernels.get_kernel("vertical_convolution")(self.queue, self.procsize[octave], self.wgsize[octave],
  File "/usr/local/lib/python3.8/dist-packages/pyopencl/__init__.py", line 887, in kernel_call
    return self._enqueue(self, queue, global_size, local_size, *args, **kwargs)
  File "<generated code>", line 8, in enqueue_knl_vertical_convolution
pyopencl._cl.MemoryError: clEnqueueNDRangeKernel failed: MEM_OBJECT_ALLOCATION_FAILURE
terminate called without an active exception
/code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12:   185 Aborted                 (core dumped) "$PYTHON" "$DIR"/opensfm_main.py "$@"

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 134
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 94, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 328, in run
    self.process(self.args, outputs)
  File "/code/stages/run_opensfm.py", line 35, in process
    octx.feature_matching(self.rerun())
  File "/code/opendm/osfm.py", line 297, in feature_matching
    self.run('detect_features')
  File "/code/opendm/osfm.py", line 34, in run
    system.run('%s %s "%s"' %
  File "/code/opendm/system.py", line 106, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 134

[Saijin-Naib]

Another user hit this:
https://community.opendronemap.org/t/odm-failed-at-opensfm-feature-stage-with-gpu/9555?u=saijin_naib

[pierotofy]

The workaround at the moment is to use HAHOG.

[pierotofy]

Plan is to replace the current GPU code with https://github.com/uav4geo/pypopsift

[rumenmitrev]

I also faced this problem even with smaller images - 4000 pixels I think is just run out of GPU memory. Fine tuning of resize-to helped me to pass to next stage of pipeline. I had to resize-to 3000.