The HyP3-autoRIFT plugin provides a set of workflows for feature tracking processing with the AutoRIFT autonomous Repeat Image Feature Tracking (autoRIFT) software package. This plugin is part of the Alaska Satellite Facility's larger HyP3 (Hybrid Plugin Processing Pipeline) system, which is a batch processing pipeline designed for on-demand processing of remote sensing data. For more information on HyP3, see the Background section.
hyp3-autorift
repository and navigate to the root directory of this project
git clone https://github.com/ASFHyP3/hyp3-autorift.git
cd hyp3-autorift
Create and activate your Python environment
mamba env create -f environment.yml
mamba activate hyp3-autorift
python -m pip install -e .
The HyP3-autoRIFT plugin provides workflows (accessible directly in Python or via a CLI) that can be used to process SAR data or optical data using autoRIFT. HyP3-autoRIFT can process these satellite missions:
To see all available workflows, run:
python -m hyp3_autorift ++help
hyp3_autorift
workflowThe hyp3_autorift
workflow is used to get dense feature tracking between two images using autoRIFT. You can run this workflow by selecting the hyp3_autorift
process:
python -m hyp3_autorift ++process hyp3_autorift [WORKFLOW_ARGS]
or by using the hyp3_autorift
console script:
hyp3_autorift [WORKFLOW_ARGS]
For example:
hyp3_autorift \
"S2B_MSIL1C_20200612T150759_N0209_R025_T22WEB_20200612T184700" \
"S2A_MSIL1C_20200627T150921_N0209_R025_T22WEB_20200627T170912"
This command will run autorift for a pair of Sentinel-2 images.
[!IMPORTANT] Credentials are necessary to access Landsat and Sentinel-1 data. See the Credentials section for more information.
For all options available to this workflow, see the help documentation:
hyp3_autorift --help
Depending on the mission being processed, some workflows will need you to provide credentials. Generally, credentials are provided via environment variables, but some may be provided by command-line arguments or via a .netrc
file.
To process Landsat images, you must provide AWS credentials because the data is hosted by USGS in a "requester pays" bucket. To provide AWS credentials, you can either use an AWS profile specified in your ~/.aws/credentials
by exporting:
export AWS_PROFILE=your-profile
or by exporting credential environment variables:
export AWS_ACCESS_KEY_ID=your-id
export AWS_SECRET_ACCESS_KEY=your-key
export AWS_SESSION_TOKEN=your-token # optional; for when using temporary credentials
For more information, please see: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/credentials.html
To process Sentinel-1 images, you must provide Earthdata Login credentials in order to download input data.
For Earthdata login, you can provide credentials by exporting environment variables:
export EARTHDATA_USERNAME=your-edl-username
export EARTHDATA_PASSWORD=your-edl-password
or via your ~/.netrc
file which should contain lines like these two:
machine urs.earthdata.nasa.gov login your-edl-username password your-edl-password
[!TIP] Your
~/.netrc
file should only be readable by your user; otherwise, you'll receive a "net access too permissive" error. To fix, run:chmod 0600 ~/.netrc
The ultimate goal of this project is to create a docker container that can run autoRIFT workflows within a HyP3 deployment. To run the current version of the project's container, use this command:
docker run -it --rm \
-e AWS_ACCESS_KEY_ID=[YOUR_KEY] \
-e AWS_SECRET_ACCESS_KEY=[YOUR_SECRET] \
-e EARTHDATA_USERNAME=[YOUR_USERNAME_HERE] \
-e EARTHDATA_PASSWORD=[YOUR_PASSWORD_HERE] \
ghcr.io/asfhyp3/hyp3-autorift:latest \
++process hyp3_autorift \
[WORKFLOW_ARGS]
[!TIP] You can use
docker run --env-file
to capture all the necessary environment variables in a single file.
To retain hyp3_autorift output files running via Docker there are two recommended approaches:
Use a volume mount
Add the -w /tmp -v ${PWD}:/tmp
flags after docker run
; -w
changes the working directory inside the container to /tmp
and -v
will mount your current working directory to the /tmp
location inside the container such that hyp3_autorift outputs are preserved locally. You can replace ${PWD}
with any valid path.
Copy outputs to a remote AWS S3 Bucket
Append the --bucket
and --bucket-prefix
to [WORKFLOW_ARGS] so that the final output files are uploaded to AWS S3. This also requires that AWS credentials to write to the bucket are available to the running container. For example, to write outputs to a hypothetical bucket s3://hypothetical-bucket/test-run/
:
docker run -it --rm \
-e AWS_ACCESS_KEY_ID=[YOUR_KEY] \
-e AWS_SECRET_ACCESS_KEY=[YOUR_SECRET] \
-e AWS_SESSION_TOKEN=[YOUR_TOKEN] \ # Optional
-e EARTHDATA_USERNAME=[YOUR_USERNAME_HERE] \
-e EARTHDATA_PASSWORD=[YOUR_PASSWORD_HERE] \
ghcr.io/asfhyp3/hyp3-autorift:latest \
++process hyp3_autorift \
[WORKFLOW_ARGS] \
--bucket "hypothetical-bucket" \
--bucket-prefix "test-run"
HyP3 is broken into two components: the cloud architecture/API that manages the processing of HyP3 workflows and Docker container plugins that contain scientific workflows that produce new science products from a variety of data sources (see figure below for the full HyP3 architecture).
The cloud infrastructure-as-code for HyP3 can be found in the main HyP3 repository., while this repository contains a plugin that can be used for feature tracking processing with AutoRIFT.
The HyP3-autoRIFT plugin is licensed under the BSD 3-Clause license. See the LICENSE file for more details.
We strive to create a welcoming and inclusive community for all contributors to HyP3-autoRIFT. As such, all contributors to this project are expected to adhere to our code of conduct.
Please see CODE_OF_CONDUCT.md
for the full code of conduct text.
Contributions to the HyP3-autoRIFT plugin are welcome! If you would like to contribute, please submit a pull request on the GitHub repository.
Want to talk about HyP3-autoRIFT? We would love to hear from you!
Found a bug? Want to request a feature? open an issue
General questions? Suggestions? Or just want to talk to the team? chat with us on gitter