Lots of STAC generation improvements

[omad]

Cleanup

• Remove lambda_modules/ dir that is no longer being used.

  STAC Lambda

• Lots of editing and clarity improvements of the README

• Use Serverless Cloudformation instead of Terraform

• Add AWS tags to serverless generated AWS resources

• Benchmark stac lambda

• Use ruamel.yaml instead of pyyaml. It includes the binary libyaml by default, for a 40x speedup of the lambda function.

• Remove hard-coded AWS bucket names and make them dynamic dependent on deployment stage.

  Manual Submission

• Use batch submission to SQS within notify_to_stac_queue.py for a 10x speedup.

• Allow stac_parent_update.py to load S3 Keys from a text file. Allows preprocessing and checking before running.

• Add a --dry-run option to stac_parent_update to allow testing and basic benchmarking.

• Rename CatalogUpdater to StacCollections and split the API into instanciate, add and then persist to finally save to S3.

• Remove update_product_suite_catalogs.py. These catalogs will be better managed manually, the code is an extra overhead.

General

• Use str.endswith instead of pathlib.Path for some URL matching, when simpler.
• Use pathlib.PurePosixPath, since Path cannot be used on Windows for manipulating URLs due to using \ as a path separator.
• Use dateutil.parser instead of pandas for a much lighter weight date parsing solution.

Tests

• Rewrite unit tests to use moto library for mocking AWS services. This makes it possible to rapidly re-run tests.
• Use a separate configuration for testing than for deployment. We want to test the code, not code+config together.

Production Ready

• Add logging throughout stac.py and notify_to_stac_queue.py. It's essential when using AWS Lambda so that it's possible to debug errors. In particular, knowing which objects have been processed, and which have been skipped.

Configuration

• Update Fractional Cover S3 prefix
• Comment out conversion of all but Fractional Cover until they can be reviewed.

[ashoka1234]

Thanks Damien. It is great. But I ran into some deployment issues. Newest pyproj module in lambda failed for me but downgrading it worked!