Overlap removal with imprint relocation

What:

Overlapping images can be detected using computer vision techniques based on the features located within the images.

Why:

Duplicates in image datasets can bias ecological analyses, as they may lead to overestimations of certain species or individuals. However, these duplicates can also provide valuable alternate views of rare morphotypes. In large datasets with more than 1,000 images, manually detecting overlaps can be time-consuming. Automated detection using computer vision offers an efficient solution.

How:

Feature matching is a method that allows a computer to detect common features in pairs of images. These common features can be validated by assessing the distance between points in candidate pairs using RANSAC (Random Sample Consensus) regression. The number of features detected is typically sufficient to establish a threshold for overlap, which can then be stored in a contingency matrix.

There’s also the possibility of using these matched points to calculate a homography matrix, which can then be used for mosaicking or cropping overlapping portions of images. The sequence of image comparison can be customized depending on the user's needs (refer to the COLMAP tutorial).

Python Code Example:

Feature Matching and Overlap Detection:

import cv2
import numpy as np

# Load the images
img1 = cv2.imread('image1.jpg')
img2 = cv2.imread('image2.jpg')

# Convert images to grayscale
gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

# Detect ORB keypoints and descriptors
orb = cv2.ORB_create()
keypoints1, descriptors1 = orb.detectAndCompute(gray1, None)
keypoints2, descriptors2 = orb.detectAndCompute(gray2, None)

# Match descriptors using BFMatcher
bf = cv2.BFMatcher(cv2.NORM_HAMMING, crossCheck=True)
matches = bf.match(descriptors1, descriptors2)

# Sort matches by distance
matches = sorted(matches, key=lambda x: x.distance)

# Calculate overlap using RANSAC
src_pts = np.float32([keypoints1[m.queryIdx].pt for m in matches]).reshape(-1, 1, 2)
dst_pts = np.float32([keypoints2[m.trainIdx].pt for m in matches]).reshape(-1, 1, 2)

homography_matrix, mask = cv2.findHomography(src_pts, dst_pts, cv2.RANSAC, 5.0)

# Calculate the number of inliers (good matches)
inliers = np.sum(mask)
print(f"Number of inliers: {inliers}")

# Determine if images overlap based on the number of inliers
overlap_threshold = 10  # This threshold can be adjusted based on your needs
is_overlapping = inliers > overlap_threshold
print(f"Images overlap: {is_overlapping}")

Creating a Contingency Matrix:

import numpy as np

# Example list of image pairs (normally this would be generated automatically)
image_pairs = [('image1.jpg', 'image2.jpg'), ('image1.jpg', 'image3.jpg'), ('image2.jpg', 'image3.jpg')]

# Initialize an empty contingency matrix
num_images = len(image_pairs)
contingency_matrix = np.zeros((num_images, num_images))

# Populate the contingency matrix
for i, (imgA, imgB) in enumerate(image_pairs):
    # Check for overlap (here using the previous is_overlapping result)
    overlap_result = is_overlapping  # This should be calculated for each pair

    # Update the matrix (1 for overlap, 0 for no overlap)
    contingency_matrix[i // num_images, i % num_images] = 1 if overlap_result else 0

print("Contingency Matrix:")
print(contingency_matrix)

Optional: Homography Matrix for Mosaicking or Cropping:

if is_overlapping:
    # Warp the second image to align with the first image using the homography matrix
    height, width, channels = img1.shape
    img2_aligned = cv2.warpPerspective(img2, homography_matrix, (width, height))

    # Combine or crop the images as needed (example for mosaicking)
    stitched_image = np.maximum(img1, img2_aligned)

    # Save or display the result
    cv2.imshow('Stitched Image', stitched_image)
    cv2.waitKey(0)
    cv2.destroyAllWindows()

What to expect:

The process will produce a contingency matrix indicating whether each pair of images overlaps (1 for overlap, 0 for no overlap). This matrix can guide further analysis, such as removing duplicates or preparing data for mosaicking. Additionally, the homography matrix calculated from feature matching can be used for mosaicking or cropping overlapping portions of images.

What makes it difficult:

Computer vision algorithms can yield variable results depending on image quality, so parameter tuning might be necessary. It's important to validate the feature-matching overlap detection by manually checking a subset of the images. This ensures that the automated process is functioning as expected and that no important details are missed.

paidiver / paidiverpy