Stac

[ashoka1234]

Reason for this pull request

STAC item and catalog files are required to be created for certain products that are uploaded to AWS s3 prod buckets. STAC files must conform to current STAC version 0.6.0. It was decided that these files are to be created/updated separate from product upload processes and needs to be automated.

Proposed Solution

The STAC item files are generated event-driven using a serverless lambda triggered by messages arriving in a AWS SQS queue. When a yaml file corresponding to a dataset is uploaded/modified into a s3 bucket, AWS is configured to send a message to this SQS queue. The parent catalog files are created/updated collectively and incrementally in batch mode. The following are the respective functions of the corresponding scripts:

• [ ] stac.py contains the lambda function to generate and upload STAC item corresponding to a dataset.
• [ ] serverless.yml contains the serverless deployment script that deploy the lambda function in stac.py
• [ ] notify_to_stac_queue.py is a script that could be run to send event notifications that are similar to typical s3 put events to SQS queue in batch mode. This script scans through AWS inventory lists to identify yaml files corresponding to datasets.
• [ ] stac_parent_update.py creates/updates STAC parent catalog files in batch mode collectively/incrementally. This script also scans through AWS inventory lists to identify yaml files corresponding to datasets.
• [ ] blow_stac_parent_catalogs.py deletes STAC parent catalog files from a s3 bucket. This script also scans through AWS inventory lists to identify catalog.json files.
• [ ] terraform script is still work in progress

[ashoka1234]

The configs now separated into a separate file stac_config.yaml.

You can test the lambda function and notify_to_stac_queue.py by running python notify_to_stac_queue.py -b dea-public-data-dev fractional-cover/fc/v2.2.0/ls8/x_-12/y_-12/2018/02/22/LS8_OLI_FC_3577_-12_-12_20180222125938.yaml

then verifying the respective STAC item file at https://s3-ap-southeast-2.amazonaws.com/dea-public-data-dev/fractional-cover/fc/v2.2.0/ls8/x-12/y-12/2018/02/22/LS8_OLI_FC3577-12_-12_20180222125938_STAC.json

Similarly you can test the script stac_parent_update.py by using a command like python stac_parent_update.py -b dea-public-data-dev fractional-cover/fc/v2.2.0/ls8/x_-12/y_-12/2018/02/22/LS8_OLI_FC_3577_-12_-12_20180222125938.yaml

and then verifying the catalog.json files at

• https://s3-ap-southeast-2.amazonaws.com/dea-public-data-dev/fractional-cover/fc/v2.2.0/ls8/catalog.json
• https://s3-ap-southeast-2.amazonaws.com/dea-public-data-dev/fractional-cover/fc/v2.2.0/ls8/x_-12/catalog.json
• https://s3-ap-southeast-2.amazonaws.com/dea-public-data-dev/fractional-cover/fc/v2.2.0/ls8/x-12/y-12/catalog.json

[omad]

Looks good Ashoka!

I've got some improvements in mind for the metadata being produced, but that can be done separately.