feat: create notebook that publishes collection and item ingest for most collections

NASA-IMPACT / veda-data

2 stars 0 forks source link

feat: create notebook that publishes collection and item ingest for most collections #111

Closed botanical closed 3 months ago

botanical commented 3 months ago

Changes

This notebook is based off of the reconciliation notebook but has been updated to:

not merge summaries and retain external links
point to MCP test and production

exclude

hls-l30-002-ej-reprocessed
hls-s30-002-ej-reprocessed
ls8-covid-19-example-data
landsat-c2l2-sr-antarctic-glaciers-pine-island
landsat-c2l2-sr-lakes-aral-sea
landsat-c2l2-sr-lakes-tonle-sap
landsat-c2l2-sr-lakes-lake-balaton
landsat-c2l2-sr-lakes-vanern
landsat-c2l2-sr-antarctic-glaciers-thwaites
landsat-c2l2-sr-lakes-lake-biwa
combined_CMIP6_daily_GISS-E2-1-G_tas_kerchunk_DEMO

smohiudd commented 3 months ago

Just confirming that these endpoints are from MCP test and prod?

if testing_mode:
    STAC_INGESTOR_API = "https://soltgnljhh.execute-api.us-west-2.amazonaws.com/"
    VEDA_STAC_API = "https://e9xlhyjfhk.execute-api.us-west-2.amazonaws.com/"
else:
    STAC_INGESTOR_API = "https://gh9ge13dej.execute-api.us-west-2.amazonaws.com/"
    VEDA_STAC_API = "https://6ouvl5o466.execute-api.us-west-2.amazonaws.com/"

smohiudd commented 3 months ago

In the last cell why is are you getting the collection?

        response = requests.get(url, headers=headers)
        response.raise_for_status()
        json_response = response.json()

botanical commented 3 months ago

In the last cell why is are you getting the collection?

        response = requests.get(url, headers=headers)
        response.raise_for_status()
        json_response = response.json()

Ooh nice catch @smohiudd ! That was remnant code from the other notebook. Let me remove that bit.

botanical commented 3 months ago

Just confirming that these endpoints are from MCP test and prod?

if testing_mode:
    STAC_INGESTOR_API = "https://soltgnljhh.execute-api.us-west-2.amazonaws.com/"
    VEDA_STAC_API = "https://e9xlhyjfhk.execute-api.us-west-2.amazonaws.com/"
else:
    STAC_INGESTOR_API = "https://gh9ge13dej.execute-api.us-west-2.amazonaws.com/"
    VEDA_STAC_API = "https://6ouvl5o466.execute-api.us-west-2.amazonaws.com/"

These are the test and prod endpoints but I can also use their domains if you prefer @smohiudd

smohiudd commented 3 months ago

Domains are just easier to distinguish but not a big deal.