Incorporate a GPU-accelerated version of Scikit-Image’s “SIFT” into RAPIDS cuCIM library

[kkgreen1964]

Is your feature request related to a problem? Please describe. One of the major bottlenecks in my end-to-end Geospatial application for rapidly geolocating aerial imagery is the ability to perform image similarity quickly. However, image similarity is dependent on how fast Feature Extraction can be performed. SIFT is considered to be the best feature extraction algorithm, and I have currently been using an OpenCV "CPU" version of SIFT.

Here is a slide that discusses how SIFT is used in many applications, and also some of the many benefits of using SIFT: The Scale Invariant Feature Transform (SIFT) - One Slide.pptx

So I wish cuCIM had a GPU-accelerated version of SIFT versus my current usage of the CPU OpenCV implementation. This would enable me to have another stage of my application that keeps the processing on the GPU device versus having to return from processing on the device to processing on the host's CPU.

Lastly, this request dovetails with my recent request to have "RANSAC" included in cuCIM, and my past request to have "match_descriptors" included in cuCIM (this request was fulfilled on October 5, 2022). The major "trifecta" benefit of having RANSAC, SIFT, and match_descriptors in cuCIM is that my desired goal of performing all aspects of Image Similarity can now be fully conducted on the GPU.

Describe the solution you'd like A successful GPU-accelerated version of Scikit-Image's "SIFT" algorithm. I am suggesting using Scikit-Image's version, since Scikit-Image contains implementations of "match_descriptors" and RANSAC.

Describe alternatives you've considered - A Novel Hybrid Feature Extraction Algorithm Since most feature extractors consist of two essential parts: 1) A feature detector, and 2) a feature descriptor. Then, the correspondence for two "best of breed" feature extraction algorithms, SURF and SIFT, have the following detector / descriptor components: • SURF = Fast Hessian detector + SURF's descriptor • SIFT = Difference of Gaussian’s detector + SIFT's descriptor

However, researchers have found that the Hessian detector is superior than the Difference of Gaussian's detector (here and here).

And, in my personal experiments, I have found the combination of the Hessian detector + SIFT's descriptor to outperform using either SURF only or SIFT only. I have also tried other feature detectors and feature descriptors (e.g., ORB, BRIEF). but none in my experience have outperformed the hybrid combination of using SURF's detector for extracting key points, followed by using SIFT's descriptor to extract the feature vectors, as shown below: • Hybrid SURF / SIFT = Fast Hessian detector (possible SURF patent?) + SIFT's descriptor (no more SIFT patent)

Here is the Hybrid SURF / SIFT repo for consideration to be incorporated in cuCIM, and is listed here.

Additional context - NVIDIA "in-house" implementation of Numba CUDA SIFT Matthew Penn (mpenn@nvidia.com) helped me in implementing a Numba CUDA SIFT kernel. So this might be an alternative path forward on starting with the Numba CUDA SIFT kernel, python scripts, and an associated Jupyter Notebook, and then convert to a Raw CuPy CUDA kernel to squeeze as much “CUDA juice” as possible 😊! We started with a pure Python implementation (no patent / copyright issues) listed here.

Additional context - Existing GPU versions of SIFT Also, here are some additional GPU versions of SIFT for the cuCIM team to consider for incorporation: • Celebrandil/cudaSift repo has CUDA SIFT package that has a compatible license. The C++ version is listed here, and the Python version is listed here. • KaliberAI / CudaSIFT was forked from Celebrandil/CudaSift, and is listed here. • Changchang Wu's SiftGPU implementation, which is now maintained by UNC is listed here. • PopSift is an implementation of the SIFT algorithm in CUDA. PopSift tries to stick as closely as possible to David Lowe's famous paper "Distinctive Image Features from Scale-Invariant Keypoints", while extracting features from an image in real-time on an NVIDIA GTX 980 Ti GPU. The repo is listed here.

[jakirkham]

cc @gigony @grlee77

[grlee77]

For a detailed technical description of the SIFT algorithm, there is this IPOL paper that the scikit-image implementation is based on: Anatomy of the SIFT Method

[kkgreen1964]

Thanks for sending this paper to me ☺!

On Wed, Nov 16, 2022 at 3:19 PM Gregory Lee @.***> wrote:

For a detailed technical description of the SIFT algorithm, there is this IPOL paper that the scikit-image implementation is based on: Anatomy of the SIFT Method https://www.ipol.im/pub/art/2014/82/

— Reply to this email directly, view it on GitHub https://github.com/rapidsai/cucim/issues/436#issuecomment-1317616957, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJQHYSISOFKL6YBU4SX5TDWIU6WJANCNFSM6AAAAAASCQAYVE . You are receiving this because you authored the thread.Message ID: @.***>