search

MAAP-Project / maap-hec-aws

2 stars 1 forks source link

Integrate stage-in and stage-out mechanisms from downsample DEM reference application into CI/CD automation. #42

Closed jjacob7734 closed 2 years ago

jjacob7734 commented 2 years ago

Integrate stage-in and stage-out mechanisms from downsample DEM reference application into CI/CD automation. The (i) destination bucket/folder, (ii) key id and secret, and CWL for stage out will be passed as parameters to the request.

The following use cases should be supported for release R2:

use case 1: from public url with no other requirements

stage_param_var1: HTTP: url: "google.com/1.bin"

use case 2: download from an S3 bucket with no credentials

stage_param_var2: S3_unsigned: url: "S3://bucket-name/key-name-public.png"

use case 3: using a private S3 bucket where the user knows the keys

stage_param_var3: S3: url: "S3://bucket-name/key-name-private.png" key: "hsdfg" secret: "jdfgha"

Definition of Done:

jplzhan commented 2 years ago

The CI/CD runner (referenced in #8) now builds stage_in and stage_out files that download and upload data for the specified use cases.

Refer to example CWLs here: https://github.com/jplzhan/artifact-deposit-repo/tree/main/jplzhan/gedi-subset/main

Stage In still needs to be tested for all remaining use cases.

MAAP Data Store use case is pending fixes from HEC team or may be otherwise removed.

Documentation is pending.

jjacob7734 commented 2 years ago

Tested Use Case 1 in downsample-landsat, Use Case 2 in downsample-dem, Use Case 3 in GEDI Subset