feature: notebook to reconcile collection metadata

botanical commented 8 months ago

Description

This PR includes a notebook called collection-metadata-reconciliation.ipynb which updates the summaries information for collections in ingestion-data/collections/ and retains the external links of those collections.

Note: this notebook does not reintegrate the retrieved summaries to the collection JSON file in this repo.

Changes Made

created a notebook for collection metadata reconciliation which entails using the collections in ingestion-data/collections as a base JSON object and including the merged summaries information and retained external links and posting that newly built JSON object to the ingest endpoint.
included a testing_mode to point to dev if True and staging otherwise

Testing Done

tested notebook with collections in dev

Additional Comments

I have not set and run the notebook's last cell which will write the updated collections to staging.

botanical commented 7 months ago

📣 The notebook was run against dev since the null stac properties fix was merged and you can view the cell output to see the collections were updated and do not have null properties. (cc @anayeaye @smohiudd )

anayeaye commented 7 months ago

this is the first time in such a long time that we haven't had a metadata error on the browser landing page AND render metadata are in! https://radiantearth.github.io/stac-browser/#/external/dev.delta-backend.com/api/stac/

smohiudd commented 7 months ago

Amazing! Well done! Also this proves that our dev ingestor is working for collection ingests at least

NASA-IMPACT / veda-data