The "Erdbeobachtungs-SAtellitendaten fürs TRockenheitsMOnitoring" (SATROMO) consists of python code with ETL functionalities to operationally generate and provide AnalysisReadyData and indices from satellite sensors using GoogleEarthEngine, GithubAction and AWS S3 / Cloudfront
Note: This project is currently in the commissioning phase and is limited for operational use.
This project aims to define automated spatial satellite products, indices, and analysis-ready datasets (ARD), as well as establish a geospatial data processing pipeline with traceable algorithms in the field of drought monitoring. The aim is to have a serverless processing chain to derive and publish satellite sensor derived products for drought monitoring.
There are two environments:
To install and set up the project, follow the instructions below:
git clone https://github.com/swisstopo/topo-satromo.git
pip install -r requirements.txt
[Optional] If you are using a virtual environment, activate it:
source <virtual-environment>/bin/activate
This step is recommended to isolate the project dependencies from your system-wide Python installation.
configuration\dev_config.py
.
You can use another configuration file for your own setup.
To choose another configuration, use a command-line argument to point towards
the configuration file contained in the For example, if you want the configuration file configuration\myconfig.py
to be used for your processing,
you can define it using the following statement:
python satromo_processor.py myconfig.py
Note that all configuration files have to be located in the configuration folder
For the use on your local machine / DEV
You need to create a folder "secrets" containing
keyfile.json
according to your Google Service Account and a private key for the service account. Ensure that service account has the Earth Engine API enabled and has been has been registered for use with Earth Engine. Then you also need to activate your GoogleDrive / Google developers console API activation for your ServiceAccount Project https://console.developers.google.com/apis/api/drive.googleapis.com/overview?project=rclone set up: To move files from GoogleDrive to your TargetDestination we use rclone. Make sure that either rclone is installed, and path to it is set .set up rclone.conf
:
[<GEEDRIVE_DEV as definend in the configuration file>]
type = drive
scope = drive
service_account_file = keyfile.json
team_drive =
[<S3_DEV as definend in configuration file>]
type = s3
provider = AWS
access_key_id = <your key>
secret_access_key = <your access key>
region = <your region>
location_constraint = <your region>
For the use with GithubAction / INT Adapt your GoogleEarthEngine and rclone credentials
keyfile.json
into itrclone.conf
into itsatromo_processor.py
and satromo_publisher.py
)For starters, we build a GithubAction & GEE python based ARD and Indices extractor for Switzerland. The SATROMO operational module is started on a pre-defined schedule using GitHub Actions.The "processor" run is using by default the dev_config.py. A specific configuration can be started with python satromo_processor.py my_config.py
. During a "processor" run, the code:
step0
: post-processed Google Earth Engine (GEE) sensor collection with selected co-registered bands, topographic correction (TOA products), cloud & terrain shadow maskstools/last_updates.csv
processing/running_tasks.csv
and an appropriate accompanying text file for each product which is split in quadrants to enable GEE exportsA pre-defined time after the "processor" run, a "publisher" run starts, assuming all exports are done. In it, the publisher process:
processing/running_tasks.csv
tools/last_updates.csv
.Edit configuration/dev_config.py rename it to your needs.
A personal collection contains a GEE collection my_new_collection
which contains postprocessed sensor data. The postprocessing is done according to the <new_file>.<new_function_name>
. E.g. The product ch.swisstopo-swisseo_vhi_v100
is based on the COPERNICUS/S2_SR_HARMONIZED
collection which has been postprocessed with step0_processor_s2_sr.generate_s2_sr_mosaic_for_single_date
and stored in the personal collection 'projects/username/assets/COL_S2_SR_HARMONIZED_SWISS'
The personal collection needs to be created in advance in the GEE GUI via Assets -> new -> Image collection
Note:if you want to ensure that other accounts can update/read the asset ( like satromo-int) you need to give them access via right click on asset which will then open https://console.cloud.google.com/iam-admin/iam?project=your-project and as well share "image collection" (like COL_S2_SR_HARMONIZED_SWISS).
Adding a new collection in the tool require to adapt the configuration file.
A new function specifically designed for this new collections should also be added in the step0_processors folder.
(If you don't need a new step0 processor function, you certainly don't need a new collection...)
Create the new function in a new file located in step0_processors folder.
In the configuration file, add the function to the configuration entry of your new collection:
The cleaning_older_than
removes asset older teh defiend days to save GEE storage.
step0: {
...
my_new_collection: {
step0_function: <new_file>.<new_function_name>,
cleaning_older_than: <days>
}
}
Products are defined with the following mandatory parameters. more can be added, based on needs for the product generation.Do mind that for all products based on the same sensor the same personal collection should be used
my_new_product = {
"prefix": <my_prefix>,
"temporal_coverage": <number of days to take into account for product generation>,
"spatial_scale_export": <spatial resolution in m>,
"product_name": <my_product_name>,
"step0_collection": <my_new_collection to be used>
...
}
A new function specifically designed for each personal collections is stored in the step0_processors folder. (If you don't need a new step0 processor function, you certainly don't need a new collection...) Create the new function in a new file located in step0_processors folder.
For cases where certain data was not computed on production for some reason (e.g., missing cloud mask etc.). These instructions are suitable for processing scenes via command line on the local machine using an INT or PROD account from GEE with a local version of the code and writing to PROD of STAC and GEE ASSETS.
Installation Install according to the instructions in the GitHub repository.
Copy MAIN It is recommended to copy/check out MAIN with Github to ensure consistency in data processing.
Objective The goal is to process a day, e.g., 2024-06-12, for a product.
Configure prod_config.py Configure prod_config.py so that in case of Windows:
Delete Assets Delete affected (damaged) assets in GEE step0_collection before recalculating step0.
Remove empty_step0.csv asset entry for date Delete all 'no candidate' entries for products in tools\empty_step0.csv if they are to be recalculated.
Set Date Set "LastSceneDate" of the respective product before the date you want to process, e.g., before 2024-06-12, in tools/last_updates.txt.
Start Venv Environment Start and activate the virtual environment:
c:
cd C:\temp\topo-satromo (as an example)
.venv\Scripts\activate
For June 12, 2024, process the products as defined in prod_config.py:
(.venv) C:\temp\topo-satromo>python satromo_processor.py prod_config.py 2024-06-12
GEE Asset creation complete and Start Export Check after about 60 minutes, if the corresponding assets (x2) are present in GEE.
(.venv) C:\temp\topo-satromo>python satromo_processor.py prod_config.py 2024-06-12
Start Publish Export Check after about 60 minutes, if the corresponding files (6x4) are present in Google Drive, or keep running the job below.
(.venv) C:\temp\topo-satromo>python satromo_publish.py prod_config.py
Update the step0_empty_assets.csv directly on GitHub PROD: https://github.com/swisstopo/topo-satromo/blob/main/tools/step0_empty_assets.csv
gee
and requests
(or similar) Python packagesContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.
If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!
git checkout -b feature/AmazingFeature
)git commit -m 'Add some AmazingFeature'
)git push origin feature/AmazingFeature
)Distributed under the BSD-3-Clause License. See LICENSE.txt
for more information.
David Oesch - david.oesch[ a t]swisstopo.ch Joan Sturm - joan.sturm[ a t]swisstopo.ch
Project Link: https://github.com/swisstopo/satromo