This PR adds the collection metadata for publishing the hand-curated landsat sample collections as well as a notebook for selecting, correcting, and publishing both collection and item metadata to the production account.
How tested
I ran the notebook included in this PR to
correct and publish the collection metadata to the test catalog
iterate over the items in the source catalog and fix metadata where possible and then publish only the valid items to the test catalog
repeated for production and verified that the item count in the production catalog is not significantly smaller than the staging catalog (but it is expected to be lower because the staging catalog had invalid records with hrefs to files that do not exist in s3)
Caveats
Many of the items in these collections had invalid classification metadata so I removed classification from the declared stac_extensions
Many items had hrefs to non-existent files, at the point when I hit this validation error I decided to stop incrementally fixing the item metadata and just publish the valid stac records to production. So the production counts in the audit are lower than staging counts but this is because invalid items were published to the staging catalog. It is unlikely that anyone will miss records with links to tifs that do not exist.
Formerly Blocked
No longer blocked but I am leaving these notes here in case we see a similar validation error in the future
Currently items cannot be published with an Asset not accessible: Forbidden error
However the role that is used by the titiler and that should be used by the ingest api is able to access these assets, for example for s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2023/001/113/LC08_L2SR_001113_20230125_20230208_02_T2/LC08_L2SR_001113_20230125_20230208_02_T2_SR_B4.TIF
What
This PR adds the collection metadata for publishing the hand-curated landsat sample collections as well as a notebook for selecting, correcting, and publishing both collection and item metadata to the production account.
How tested
I ran the notebook included in this PR to
Caveats
classification
metadata so I removed classification from the declared stac_extensionsFormerly Blocked
Currently items cannot be published with an
Asset not accessible: Forbidden
errorHowever the role that is used by the titiler and that should be used by the ingest api is able to access these assets, for example for
s3://usgs-landsat/collection02/level-2/standard/oli-tirs/2023/001/113/LC08_L2SR_001113_20230125_20230208_02_T2/LC08_L2SR_001113_20230125_20230208_02_T2_SR_B4.TIF