Transfer reprocessed HLS environmental justice granules and full provider metadata to production

[anayeaye]

What

We need to transfer a custom subset of HLS granules from two collections in the staging version of veda-backend to production. These collections include detailed provider metadata and have complex storage paths in S3 that require this work to be done by hand (the flat, minimal, structure of the datasets pipelines used for many veda collections would cause us to lose the detailed structure of these granules and metadata which are required for multi band maps and analyses like those used in this notebook: https://nasa-impact.github.io/veda-docs/notebooks/quickstarts/hls-visualization.html)

Background: These collections include a small number of tiles that were selected to spotlight before and after imagery from two environmental justtice stories and demonstrate how titiler+pgstac can be used to investigate that data in the cloud. Early HLS (harmonized landsat sentinel) data were selected for reprocessing (CRS data were missing from these older files). The data were downloaded from the host DAAC for reprocessing so we need to copy this set of reprocessed files to veda-data-store and then we need to update the stac metadata to href the new veda-data-store location (much more detailed than our in-house dataset metadata so we do not want to execute the build-stac pipeline for these).

Historical VEDA HLS issues

• https://github.com/NASA-IMPACT/veda-data-pipelines/issues/147
• https://github.com/NASA-IMPACT/veda-data-pipelines/issues/89

1. Transfer the files

• [x] Copy the complete contents of the HLS landsat reprocessed granules from s3://covid-eo-data/hlsl30-ej-reprocessed/ to s3://veda-data-store/hlsl30-ej-reprocessed/ without modifying the detailed key paths. I.e. s3://covid-eo-data/hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017157T144341.v2.0/HLS.L30.T19QHA.2017157T144341.v2.0.B01.tif will go to s3://veda-data-store/hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017157T144341.v2.0/HLS.L30.T19QHA.2017157T144341.v2.0.B01.tif
• [x] Do the same for the Sentinel reprocessed granules from s3://covid-eo-data/hlss30-ej-reprocessed to s3://veda-data-store/hlss30-ej-reprocessed

2. Update the hrefs in stac item records with a python script or notebook that can be reused if needed

• [x] Select all of the stac items from the hlsl30-ej-reprocessed collection and iterate over the assets to update the hrefs (now covid-eo-data will be replaced with veda-data-store will be replaced in every asset href). Publish these updated items. Alternatively, the objects copied in the first step include stac metadata that would be nice to update and use in this process (for example s3://covid-eo-data/hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017193T150719.v2.0/HLS.S30.T19QHA.2017193T150719.v2.0_stac-ej-reprocessed.json is a complete stac item record describing all band assets and thumbnails but the asset hrefs are covid-eo-data).
• [x] Do the same for sentinel series hlss30-ej-reprocessed

3. Cleanup reprocessed environmental justice story config in veda-data

• [x] Ensure that collection metadata is correct for hlsl30-ej-reprocessed and hlss30-ej-reprocessed (should be already)
• [x] Remove any airflow/DAG configuration for these collections to prevent future confusion
• [x] Copy an artifact of how the metadata were updated to veda-data//transformation-scripts to make the href update process reproducible
• [x] Process to walk HLS buckets in veda-data store and select appropriate stac-metadata files (there are multiple versions) and post those items to stac-api/ingestions endpoint

[anayeaye]

Step 1 completed and verified in veda-data-store s3 console by (first I ran with --dryrun)

• aws s3 cp s3://covid-eo-data/hlss30-ej-reprocessed s3://veda-data-store/hlss30-ej-reprocessed --recursive --dryrun
• aws s3 cp s3://covid-eo-data/hlsl30-ej-reprocessed s3://veda-data-store/hlsl30-ej-reprocessed --recursive --dryrun

[botanical]

• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/create_build_task_definition/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/ingest-slice/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/add_gitflow_enforcer/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/vector-ingest/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/change_base_tf_module/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/change_refresh_command/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/format-fixes/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/fix_black_linter/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/t_end_selection/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/deploy_to_production/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/fix/aws-provider-version/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/upgrade-rio-stac/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/lf_archival_changes/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/production/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/feature/increase_vector_task_resources/data/collections
• https://github.com/NASA-IMPACT/veda-data-airflow/tree/fix/vector-ecs-timeout/data/collections

so far all of these stale branches have the HLSL30.002.json and HLSS30.002.json files but the head branch dev does not have the data directory so we're probably fine to check off Remove any airflow/DAG configuration for these collections to prevent future confusion

[anayeaye]

I accidentally copied over the old/original stac metadata in addition to the metadata for the reprocessed assets (CRS adjustment). Old/outdated metadata jsons end as stac-ej.json, new and correct metadata are in json files named **stac-ej-reprocessed.json Here is how I cleaned out the old metadata from the veda-data-store bucket:

aws s3 rm s3://veda-data-store/hlss30-ej-reprocessed --recursive --exclude "*" --include "*stac-ej.json" --dryrun
aws s3 rm s3://veda-data-store/hlsl30-ej-reprocessed --recursive --exclude "*" --include "*stac-ej.json" --dryrun

[botanical]

The following collections were updated:

Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017157T144341.v2.0/HLS.L30.T19QHA.2017157T144341.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017173T144347.v2.0/HLS.L30.T19QHA.2017173T144347.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017205T144356.v2.0/HLS.L30.T19QHA.2017205T144356.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017221T144403.v2.0/HLS.L30.T19QHA.2017221T144403.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017237T144407.v2.0/HLS.L30.T19QHA.2017237T144407.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017269T144414.v2.0/HLS.L30.T19QHA.2017269T144414.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017285T144419.v2.0/HLS.L30.T19QHA.2017285T144419.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017301T144420.v2.0/HLS.L30.T19QHA.2017301T144420.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017317T144417.v2.0/HLS.L30.T19QHA.2017317T144417.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2017/19QHA/HLS.L30.T19QHA.2017333T144411.v2.0/HLS.L30.T19QHA.2017333T144411.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021182T163159.v2.0/HLS.L30.T15RYP.2021182T163159.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021198T163201.v2.0/HLS.L30.T15RYP.2021198T163201.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021230T163215.v2.0/HLS.L30.T15RYP.2021230T163215.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021246T163220.v2.0/HLS.L30.T15RYP.2021246T163220.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021262T163223.v2.0/HLS.L30.T15RYP.2021262T163223.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021278T163229.v2.0/HLS.L30.T15RYP.2021278T163229.v2.0_stac-ej-reprocessed.json
Updated hlsl30-ej-reprocessed/2021/15RYP/HLS.L30.T15RYP.2021294T163232.v2.0/HLS.L30.T15RYP.2021294T163232.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017193T150719.v2.0/HLS.S30.T19QHA.2017193T150719.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017218T150721.v2.0/HLS.S30.T19QHA.2017218T150721.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017233T150719.v2.0/HLS.S30.T19QHA.2017233T150719.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017278T150721.v2.0/HLS.S30.T19QHA.2017278T150721.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017293T150709.v2.0/HLS.S30.T19QHA.2017293T150709.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017318T150721.v2.0/HLS.S30.T19QHA.2017318T150721.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017323T150709.v2.0/HLS.S30.T19QHA.2017323T150709.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2017/19QHA/HLS.S30.T19QHA.2017333T150709.v2.0/HLS.S30.T19QHA.2017333T150709.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021185T163839.v2.0/HLS.S30.T15RYP.2021185T163839.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021195T163839.v2.0/HLS.S30.T15RYP.2021195T163839.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021200T163901.v2.0/HLS.S30.T15RYP.2021200T163901.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021205T163839.v2.0/HLS.S30.T15RYP.2021205T163839.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021210T163901.v2.0/HLS.S30.T15RYP.2021210T163901.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021220T163901.v2.0/HLS.S30.T15RYP.2021220T163901.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021225T163839.v2.0/HLS.S30.T15RYP.2021225T163839.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021235T163839.v2.0/HLS.S30.T15RYP.2021235T163839.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021245T163839.v2.0/HLS.S30.T15RYP.2021245T163839.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021250T163901.v2.0/HLS.S30.T15RYP.2021250T163901.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021265T163909.v2.0/HLS.S30.T15RYP.2021265T163909.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021270T164051.v2.0/HLS.S30.T15RYP.2021270T164051.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021280T164201.v2.0/HLS.S30.T15RYP.2021280T164201.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021285T164139.v2.0/HLS.S30.T15RYP.2021285T164139.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021290T164311.v2.0/HLS.S30.T15RYP.2021290T164311.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021295T164339.v2.0/HLS.S30.T15RYP.2021295T164339.v2.0_stac-ej-reprocessed.json
Updated hlss30-ej-reprocessed/2021/15RYP/HLS.S30.T15RYP.2021300T164411.v2.0/HLS.S30.T15RYP.2021300T164411.v2.0_stac-ej-reprocessed.json

I hand picked a few to check the hrefs were correctly updated and they were.

[botanical]

I created a notebook to validate that the collections were valid and marked that AC as done given that they are: https://github.com/NASA-IMPACT/veda-data/compare/jt/validation-notebook?expand=1

[anayeaye]

I re-opened this issue because there is one last step for these reprocessed HLS data: publish to mcp-test and prod

[anayeaye]

https://github.com/NASA-IMPACT/veda-data/pull/142