Shutting down Heroku instance

cisaacstern commented 11 months ago

Following https://github.com/pangeo-forge/pangeo-forge.org/pull/430, I'll be shutting down the Heroku instance.

Here are some related to-dos:

[x] Export existing catalog, and migrate it to some other public location ✅ https://github.com/leap-stc/data-management/pull/74/files
[x] Export a snapshot of the whole database (it's relatively small, I think) ✅ #239

cisaacstern commented 11 months ago

Export existing catalog, and migrate it to some other public location

Here's a glance at what's cataloged currently in the api:

from dataclasses import dataclass
import requests

api = "https://api.pangeo-forge.org"
feedstocks = requests.get(api + "/feedstocks").json()
datasets = {}
for f in feedstocks:
    dss = requests.get(f"{api}/feedstocks/{f['id']}/datasets?type=production").json()
    if dss:
        datasets.update({f["spec"]: dss})

@dataclass
class Dataset:
    recipe_id: str
    dataset_url: str

typed_datasets = {
    k: [
        Dataset(ds["recipe_id"], ds["dataset_public_url"])
        for ds in v
    ]
    for k, v in datasets.items()
}

for k, v in typed_datasets.items():
    print(k)
    for ds in v:
        print(f"  - {ds.recipe_id}: {ds.dataset_url}")

# stylizing this output as yaml for readability

pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock
  - noaa-coastwatch-geopolar-sst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr
pangeo-forge/WOA_1degree_monthly-feedstock
  - woa18-1deg-monthly: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/WOA_1degree_monthly-feedstock/woa18-1deg-monthly.zarr
pangeo-forge/cmip6-feedstock
  - test_surface: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_surface.zarr
  - test_full_depth: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_full_depth.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr
  - CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
pangeo-forge/CMIP6_static_grids-feedstock
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
pangeo-forge/gpcp-feedstock
  - gpcp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr
pangeo-forge/HadISST-feedstock
  - hadisst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/HadISST-feedstock/hadisst.zarr
pangeo-forge/AGDC-feedstock
  - AGCD: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/AGDC-feedstock/AGCD.zarr
pangeo-forge/CMIP6-PMIP-feedstock
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120.zarr
  - CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318.zarr
pangeo-forge/GPM_3IMERGDL-feedstock
  - gpm3imergdl: s3://yuvipanda-test1/cmr/gpm3imergdl.zarr/
pangeo-forge/eVolv2k-feedstock
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock
  - LMRv2p1_MCruns_ensemble_gridded: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock/LMRv2p1_MCruns_ensemble_gridded.zarr
pangeo-forge/EOBS-feedstock
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr
pangeo-forge/liveocean-feedstock
  - liveocean: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/liveocean-feedstock/liveocean.zarr
pangeo-forge/aws-noaa-oisst-feedstock
  - aws-noaa-oisst-avhrr-only: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/aws-noaa-oisst-feedstock/aws-noaa-oisst-avhrr-only.zarr

cisaacstern commented 11 months ago

Continuing with python session from previous comment, looks like all but the following three datasets are openable:

import warnings
import xarray as xr

for k, v in typed_datasets.items():
    for ds in v:
        try:
            with warnings.catch_warnings():
                warnings.simplefilter('ignore')  # ignore xarray time decode warnings
                _ = xr.open_dataset(ds.dataset_url, engine="zarr")
        except:
            print(k, ds.recipe_id)

pangeo-forge/GPM_3IMERGDL-feedstock gpm3imergdl
pangeo-forge/liveocean-feedstock liveocean
pangeo-forge/aws-noaa-oisst-feedstock aws-noaa-oisst-avhrr-only

cisaacstern commented 11 months ago

So the openable datasets are:

openable_datasets = {
    k: v
    for k, v in typed_datasets.items()
    if k not in [
         "pangeo-forge/GPM_3IMERGDL-feedstock",
         "pangeo-forge/liveocean-feedstock",
         "pangeo-forge/aws-noaa-oisst-feedstock",
    ]
}
len(openable_datasets) # -> 11

for k, v in openable_datasets.items():
    print(k)
    for ds in v:
    print(f"  - {ds.recipe_id}: {ds.dataset_url}")

pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock
  - noaa-coastwatch-geopolar-sst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/noaa-coastwatch-geopolar-sst-feedstock/noaa-coastwatch-geopolar-sst.zarr
pangeo-forge/WOA_1degree_monthly-feedstock
  - woa18-1deg-monthly: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/WOA_1degree_monthly-feedstock/woa18-1deg-monthly.zarr
pangeo-forge/cmip6-feedstock
  - test_surface: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_surface.zarr
  - test_full_depth: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/test_full_depth.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.CMIP.CCCma.CanESM5.historical.r1i1p1f1.Omon.zos.gn.v20190429.zarr
  - CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.NOAA-GFDL.GFDL-ESM4.hist-aer.r1i1p1f1.Amon.pr.gr1.v20180701.zarr
  - CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.BCC.BCC-CSM2-MR.hist-aer.r1i1p1f1.Amon.pr.gn.v20190507.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r13i1p2f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r10i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.DAMIP.CCCma.CanESM5.hist-aer.r3i1p1f1.Amon.pr.gn.v20190429.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/cmip6-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
pangeo-forge/CMIP6_static_grids-feedstock
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - NorESM2-MM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-MM.zarr
  - GFDL-ESM4: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/GFDL-ESM4.zarr
  - NorESM2-LM: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/NorESM2-LM.zarr
  - MPI-ESM1-2-LR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-LR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
  - MPI-ESM1-2-HR: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/CMIP6_static_grids-feedstock/MPI-ESM1-2-HR.zarr
pangeo-forge/gpcp-feedstock
  - gpcp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr
pangeo-forge/HadISST-feedstock
  - hadisst: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/HadISST-feedstock/hadisst.zarr
pangeo-forge/AGDC-feedstock
  - AGCD: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/AGDC-feedstock/AGCD.zarr
pangeo-forge/CMIP6-PMIP-feedstock
  - CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MPI-M.MPI-ESM1-2-LR.past2k.r1i1p1f1.Amon.tas.gn.v20210714.zarr
  - CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MRI.MRI-ESM2-0.past1000.r1i1p1f1.Amon.tas.gn.v20200120.zarr
  - CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/CMIP6-PMIP-feedstock/CMIP6.PMIP.MIROC.MIROC-ES2L.past1000.r1i1p1f2.Amon.tas.gn.v20200318.zarr
pangeo-forge/eVolv2k-feedstock
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
  - eVolv2k: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/eVolv2k-feedstock/eVolv2k.zarr
pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock
  - LMRv2p1_MCruns_ensemble_gridded: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/LMRv2p1_MCruns_ensemble_gridded-feedstock/LMRv2p1_MCruns_ensemble_gridded.zarr
pangeo-forge/EOBS-feedstock
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr
  - eobs-wind-speed: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-wind-speed.zarr
  - eobs-surface-downwelling: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-surface-downwelling.zarr
  - eobs-tg-tn-tx-rr-hu-pp: https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/EOBS-feedstock/eobs-tg-tn-tx-rr-hu-pp.zarr

cisaacstern commented 11 months ago

Moving towards a PR to the leap data catalog :

import yaml

@dataclass
class FlatDataset:
    feedstock: str
    recipe_id: str
    dataset_url: str

flat_datasets = []
for k, v in openable_datasets.items():
    for ds in v:
        flat_datasets.append(
            FlatDataset(k, ds.recipe_id, ds.dataset_url)
        )

@dataclass
class Store:
    name: str
    href: str

@dataclass
class Maintainer:
    name: str
    github_username: str

@dataclass
class LEAPCatalogEntry:
    name: str
    description: str
    stores: list[Store]
    maintainers: list[Maintainer]

leap_catalog_entries = []

for ds in flat_datasets:
    meta_text = requests.get(
        f"https://raw.githubusercontent.com/{ds.feedstock}/main/feedstock/meta.yaml"
    ).text
    meta_yaml = yaml.safe_load(meta_text)

    maintainers = [Maintainer(m["name"], m["github"]) for m in meta_yaml["maintainers"]]
    store = Store(name="Pangeo Forge Open Storage Network (OSN)", href=ds.dataset_url)

    lce = LEAPCatalogEntry(
        name=ds.recipe_id,
        description=meta_yaml["description"],
        stores=[store],
        maintainers=maintainers,
    )
    leap_catalog_entries.append(lce)

cisaacstern commented 11 months ago

Oops I just realized that I had the structure of the LEAP Catalog wrong, this should be closer to the mark:

leap_catalog_entries_2 = []

for feedstock, datasets in openable_datasets.items():
    meta_text = requests.get(
        f"https://raw.githubusercontent.com/{feedstock}/main/feedstock/meta.yaml"
    ).text
    meta_yaml = yaml.safe_load(meta_text)

    maintainers = [Maintainer(m["name"], m["github"]) for m in meta_yaml["maintainers"]]

    stores = []
    for ds in datasets:
        stores.append(Store(name=ds.recipe_id, href=ds.dataset_url))

    lce = LEAPCatalogEntry(
        name=meta_yaml["title"],
        description=meta_yaml["description"],
        stores=stores,
        maintainers=maintainers,
    )
    leap_catalog_entries_2.append(lce)

cisaacstern commented 11 months ago

Just deleted the Heroku deployment from the Heroku dashboard, so this is resolved!

pangeo-forge / pangeo-forge-orchestrator

Shutting down Heroku instance #238