search

NASA-IMPACT / veda-data-airflow

Airflow implementation of ingest pipeline for VEDA STAC data
Other
10 stars 4 forks source link

Use SM2A endpoint in Eventbridge Lambda function #248

Closed smohiudd closed 3 weeks ago

smohiudd commented 1 month ago

Description

The Eventbridge Lambda that triggers EIS vector ingests uses MWAA authorization and endpoints. To support the eventual cutover to SM2A, the function must be modified to support SM2A endpoints.

Acceptance Criteria

anayeaye commented 1 month ago

@amarouane-ABDELHAK I will set up these prefixes in VEDA-Prod for testing. In the event bridge lambda can you change these prefixes for testing? I still plan to look more closely at the fire detection ingest events that are successfully being processed but I think these are the only changes we will need in the dev lambda

Around line 25 bucket_key_prefix = "EIS/FEDSoutput/Snapshot/" --> bucket_key_prefix = "EIS/FEDSoutput/Snapshot_COPY/"

Around line 28 bucket_key_prefix = "EIS/FEDSoutput/LFArchive/" --> bucket_key_prefix = "EIS/FEDSoutput/LFArchive_COPY/"

anayeaye commented 1 month ago

@amarouane-ABDELHAK I opened a branch to start updating the s3 event bridge lambda for SM2A (https://github.com/NASA-IMPACT/veda-data-airflow/compare/feat/use-sm2a-in-eventbrige-lambda) but didn't get far. I will read up on the client and auth changes needed. If you get a chance next week, though, I suspect the changes are probably pretty obvious to you already.

amarouane-ABDELHAK commented 4 weeks ago

I am planning to work on this today

anayeaye commented 3 weeks ago

bucket event properties I copied all 4 veda-data-store-staging events (2 .fgb and 2 .gpkg) and updated the bucket to the new copied prefixes (i.e. _COPY).

files to copy I think we can just copy the newest files when we want to trigger events

#!/bin/bash

SOURCE_BUCKET="veda-data-store-staging"
PREFIX="EIS/FEDSoutput/Snapshot/"
DESTINATION_PREFIX="EIS/FEDSoutput/Snapshot_COPY/"

content=$(aws s3api list-objects-v2 --bucket $SOURCE_BUCKET --query 'Contents[?(LastModified > `2024-10-28`)]' --prefix $PREFIX | jq -r ".[].Key")

for file in $content;
do
    export DESTINATION_KEY=${file//$PREFIX/$DESTINATION_PREFIX}
    echo ${DESTINATION_KEY}
    echo aws s3 cp $SOURCE_BUCKET/$file $SOURCE_BUCKET/$DESTINATION_KEY
done
# do
#     export DESTINATION_KEY=${file//$PREFIX/$DESTINATION_PREFIX}
#     aws s3 cp $SOURCE_BUCKET/$file $SOURCE_BUCKET/$DESTINATION_KEY
# done

PREFIX="EIS/FEDSoutput/LFArchive/"
DESTINATION_PREFIX="EIS/FEDSoutput/LFArchive_COPY/"
for file in $content;
do
    export DESTINATION_KEY=${file//$PREFIX/$DESTINATION_PREFIX}
    echo ${DESTINATION_KEY}
    echo aws s3 cp $SOURCE_BUCKET/$file $SOURCE_BUCKET/$DESTINATION_KEY
done
# do
#     export DESTINATION_KEY=${file//$PREFIX/$DESTINATION_PREFIX}
#     aws s3 cp $SOURCE_BUCKET/$file $SOURCE_BUCKET/$DESTINATION_KEY
# done

dev features api to watch