Reprojection with pitch/roll and Altitude correction

What:

Reprojection of an image involves estimating a correction and applying a real-world scaling to the image by tracing the path of light rays. This process adjusts for distortions and ensures that the image accurately reflects real-world dimensions and geometry.

Why:

Reprojection corrects images for distortions, enabling accurate computation of areas and distances within the image. Compared to traditional methods like manual annotation of ground truth scales or mosaic computations, reprojection can be more time-efficient and precise, especially in complex environments.

How:

Reprojection involves tracing the path of light rays from the camera or image corners back to a planar surface, adjusting for various factors such as the camera’s altitude, vertical and horizontal angles of acceptance, and environmental conditions like water refraction.

To perform reprojection, the user must provide key parameters such as:

Altitude (e.g., the height of the camera above the seabed in underwater environments)
Vertical and horizontal angles of acceptance (defining the field of view)
Roll, pitch, pan, and tilt of the camera (to account for its orientation)
Refractive index of the medium (important for underwater environments, typically ( n = 1.30-1.35 ) for seawater)

These parameters help in constructing a mathematical model to correct the image.

Calibration of the refractive index with ground truthing in ‘real conditions’ is therefore highly recommended:

Equation 1

Below are examples of how this can be implemented in Python.

Python Code Example:

Basic Reprojection Setup:

import numpy as np
import cv2

# Define camera parameters
altitude = 10  # meters above the seabed
vertical_angle = np.radians(30)  # in radians
horizontal_angle = np.radians(40)  # in radians
roll, pitch, yaw = np.radians([5, 2, 0])  # in radians

# Image dimensions
img_width, img_height = 1920, 1080

# Compute the homography matrix (example with identity, replace with actual computation)
H = np.eye(3)

# Apply the homography to the image
image = cv2.imread('path_to_your_image.jpg')
reprojected_image = cv2.warpPerspective(image, H, (img_width, img_height))

# Display the reprojected image
cv2.imshow('Reprojected Image', reprojected_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Incorporating Roll, Pitch, and Refraction:

# Adjusting for camera roll, pitch, and refraction index
def apply_reprojection(image, H, roll, pitch, refraction_index):
    # Adjust homography for roll and pitch
    R_roll = np.array([[1, 0, 0],
                       [0, np.cos(roll), -np.sin(roll)],
                       [0, np.sin(roll), np.cos(roll)]])

    R_pitch = np.array([[np.cos(pitch), 0, np.sin(pitch)],
                        [0, 1, 0],
                        [-np.sin(pitch), 0, np.cos(pitch)]])

    # Combine transformations
    H_adj = H @ R_roll @ R_pitch

    # Adjust for refraction
    H_adj[2, 2] *= refraction_index

    # Apply adjusted homography
    reprojected_image = cv2.warpPerspective(image, H_adj, (img_width, img_height))
    return reprojected_image

# Apply the reprojection
reprojected_image = apply_reprojection(image, H, roll, pitch, refraction_index=1.33)

# Display the reprojected image
cv2.imshow('Reprojected Image with Adjustments', reprojected_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

What to expect:

The reprojection process will generate a 3x3 homography matrix, which can be applied to any image coordinate (e.g., image pixels or annotation coordinates). This matrix allows the image to be corrected for distortions caused by pan, tilt, roll, pitch, and environmental factors like water refraction, making it more accurate for subsequent analysis, such as area computation and annotations.

What makes it difficult:

This process can be demanding in terms of metadata, requiring detailed knowledge of camera parameters, environmental conditions, and precise modeling to achieve accurate results. The complexity increases in environments like underwater, where factors such as light refraction significantly affect the reprojection accuracy.

paidiver / paidiverpy