CoastalLens: A MATLAB UAV Video Stabilization & Rectification Framework
Software to stabilize and rectify coastal imagery UAV data.
Developed from the CIRN Qualitative Coastal Imagining Toolbox.
Uncrewed aerial vehicles (UAVs) are an important tool for coastal monitoring with their relatively low-cost and rapid deployment capabilities. To generate scientific-grade image products, the UAV images/videos must be stabilized and rectified into world coordinates. Due to the limited stable region of coastal images suitable for control points, the processing of UAV-obtained videos can be time-consuming and resource-intensive. The CIRN Qualitative Coastal Imagining Toolbox provided a first-of-its-kind open-sourced code for rectifying these coastal UAV videos. Limitations of the toolbox, however, prompted the development of CoastalLens with an efficient data input procedure, providing capabilities to obtain drone position (extrinsics) from LiDAR surveys, and using a feature detection and matching algorithm to stabilize the video prior to rectification. This framework reduces the amount of human oversight, now only required during the data input processes. Removing the dependency on threshold stability control points can also result in less time in the field. We hope this framework will allow for more efficient processing of the ever-increasing coastal UAV datasets.
Installation
Requires MATLAB (min v2022b - for estworldpose function, see Input Requirements for an alternative if using an older MATLAB version). Ubuntu users: See Issue 11
Required MATLAB toolboxes:
- Image Processing Toolbox
- Computer Vision Toolbox
- LiDAR Toolbox (to use pointcloud).
Requires exiftool (or metadata csv file : See Input Requirements)
Requires ffmpeg.
See Installation Help for installation guides based on your OS.
Running the Toolbox
Download or clone the repository.
Run 'UAV_rectification.m'
Will run core scripts in CODES/scripts/
Requires dependencies in CODES/basicFunctions and CODES/helperFunctions
Running ver in the Command Window will show your MATLAB version and installed toolboxes.
You can find more details, including the information you need to run this on your own data in our wiki!
Demo / Getting Started
'UAV_rectification_DEMO.m' runs a demo version of the code and can also be used to compare the new algorithm versus the CIRN Stability Control Points method. This method requires stability control points to be visible within the field of view.
Data to test the code is provided in the DATA folder and the video can be downloaded here (716MB). Save this video in DATA/20211215_Torrey/Flight_04/.
Here is the input information required for the DEMO version. We recommend making a similar table to keep track of all the necessary information for your own drone flights.
Flight information:
Origin information:
Products information:
User prompts/direction is printed in the Command Window.
Testing
This toolbox is currently in testing phase on the following systems:
- MacBook Pro M1 2020 (OS 12.6), Matlab 2022b
- MacBook Pro M2 2023 (OS 13.2.1), Matlab 2023a
- Linux (Ubuntu 22.04.3 LTS), Matlab 2022b
- DJI Drones
Email Updates
The code allows you to recieve email updates as it processes the data. If you do not want to recieve these, please select 'No' to 'Recieve update emails?'.
If you do, we have set up a Gmail account 'coastallens1903' to use that will be sending the emails, although we recommend you setting up your own account and generating a static App password (16-character) for it moving forward to avoid any security risks.
General Folder Structure:
Please set up your CODES and DATA folder in the following structure. The DATA folder may be located in a different folder than your general path, but must be organized as indicated, with all flights in the relevant day/location folder.
.
├── CODES
│ ├── scripts
│ ├── basicFunctions
│ ├── helperFunctions
├── DATA
│ └── YYYYMMDD_Location1
│ ├── Flight_01
│ ├── Flight_02
│ ├── Flight_03
│ └── YYYYMMDD_Location2
│ ├── Flight_01
│ ├── Flight_02
│ ├── Flight_03Core Scripts
| Scripts | Description |
|---|---|
UAV_rectification |
The main code. Used to rectify and generate data products for user-selected days. |
input_day_flight_data |
input_day_flight_data returns all user-specified required input data for CoastalLens. |
extract_images_from_UAV |
extract_images_from_UAV extracts images from video files at specified frame rates for all flights on specified processing days. Requires ffmpeg. |
stabilize_video |
stabilize_video returns the 2D projective transformation of the image to improve image stabilization through flight. |
get_products |
get_products returns extracted image pixel for coordinates of Products and saves Timex, Brightest and Darkest image products. |
save_products |
save_products saves rectified image products from Products in Rectified_images folder. |
Data Output
| Variable | Fields | Description |
|---|---|---|
R (structure) |
extrinsics & intrinsics information (stored in *_IOEO_*Hz.mat) | |
intrinsics(cameraIntrinsics) |
camera intrinsic as calibrated in the cameraCalibrator tool | |
I (uint8 image) |
undistorted initial frame | |
image_gcp (double) |
[n x 2] ground control location in inital frame | |
world_gcp (double) |
[n x 3] ground control location in world coordinate frame (x,y,z) | |
worldPose(rigidtform3d) |
orientation and location of camera in world coordinates, based off ground control location (pose, not extrinsic) | |
mask(logical) |
mask over ocean region (same dimension as I) - used to speed up computational time (optional) | |
feature_method(string) |
feature type to use in feature detection algorithm (default: `SIFT`, must be `SIFT`, `SURF`, `BRISK`, `ORB`, `KAZE`) | |
frameRate(double) |
frame rate of extrinsics (Hz) | |
extrinsics_2d(projtform2d) |
[1 x m] 2d projective transformation of m images. | |
Products(structure) | Data Products (stored in *_Products.mat) | |
productType(string) |
'cBathy' , 'Timestack', 'yTransect' | |
type (string) |
'Grid', 'xTransect', 'yTransect' | |
frameRate (double) |
frame rate of product (Hz) | |
lat (double) |
latitude of origin grid | |
lon (double) |
longitude of origin grid | |
angle (double) |
shorenormal angle of origin grid (deg CW from North) | |
xlim / ylim (double) |
cross-/along-shore limits (+ is offshore of origin / right of origin looking offshore) (m) | |
dx/dy (double) |
Cross-/along-shore resolution (m) | |
x / y (double) |
Cross-/along-shore distance from origin (m). Used for transects. | |
z (double) |
Elevation (m in standard reference frame). Can be NaN (will be projected to 0)or DEM. | |
tide (double) |
Tide level (m in standard reference frame). | |
t (datetime array) |
[1 x m] datetime of images at given extraction rate in UTC. | |
localX / localY / localZ (double) |
X,Y,Z coordinates of data product in local reference frame (m) | |
Eastings / Northings (double) |
Eastings and Northings coordinates of data product (m) | |
Irgb_2d (uint8 image) |
[m x y_length x x_length x 3] timeseries of pixels extracted according to dimensions of xlim and ylim |
Known Bugs
- Estimating the world camera pose based on from 3-D to 2-D point correspondences can sometime require different combinations of points. More points are not always better. We are looking into ways to improve the robustness of this step.
Contributing
Contributions to the toolbox are very welcome! Here are some ways to do that:
- A number of features that we want to include in the future are listed as 'issues'.
- We also want to make sure that we can accommodate other UAV platforms/locations. If you include any changes to be able to process your data in a forked branch, please open a 'pull request' to merge it so other's can also use the addition.
- One of our goals is to write a Python version of this toolbox, so please let us know if you are interested.
- Let us know of any other features you would want included.
If you run into any problems while running the code, or think other things should be included, please let us know by opening an 'issue' or emailing me at alange@ucsd.edu.
License
CoastalLens is provided under the MIT license.
Cite As
A.M.Z. Lange, H. Lange, B.L. Bruder and J.W. Fiedler "CoastalLens: A MATLAB UAV Video Stabilization & Rectification Framework" (2024)
