Index the Global Change Master Directory: repositories that contain more than just polar data

[yemoski]

GCMD

Top priority relevant repositories that only contain polar data

• [x] AADC (?data_center=AU/AADC)
• [x] World Glacier Monitoring Service, Zurich ?data_center=WGMS
• [x] NOAA National Centers for Environmental Information (NCEI) ?data_center=DOC/NOAA/NESDIS/NCEI
• [x] NSIDC DAAC ?data_center=NASA NSIDC DAAC
• [x] Alaska Satellite Facility ?data_center=ASF
• [x] Norwegian Marine Data Centre (NMD) ?data_center=NO/NMDC/IMR
• [x] Permanent Service for Mean Sea Level (PSMSL) ?data_center=UK/NERC/POL/PSMSL
• [x] WDC - Geomagnetism, Edinburgh ?data_center=WDC/GEOMAGNETISM, EDINBURGH
• [x] WDC for Solid Earth Physics, Moscow ?data_center=WDC/SEP, MOSCOW

Relevant repositories that contain data that needed to be scoped down to polar data

• [ ] Flanders Marine Institute, Data Centre ?data_center=BE/VLIZ
• [ ] WDC - Oceanography, Obninsk ?data_center=WDC/OCEANOGRAPHY, OBNINSK
• [ ] Oak Ridge National Laboratory Distributed Active Archive Center (ORNL DAAC) ?data_center=ORNL_DAAC
• [ ] WDC - Earth Resources Observation and Science (EROS) ?data_center=DOI/USGS/EROS
• [ ] Crustal Dynamics Data Information System (CDDIS) ?data_center=NASA/GSFC/SED/ESD/GGL/CDDIS
• [ ] World Data Service for Paleoclimatology ?data_center=WDC/PALEOCLIMATOLOGY, BOULDER
• [ ] Goddard Earth Sciences Data and Information Services Center (GES DISC) ?data_center=NASA/GSFC/SED/ESD/GCDC/GESDISC
• [ ] Atmospheric Science Data Center (Distributed Active Archive Center) ?data_center=NASA/LARC/SD/ASDC
• [ ] Socioeconomic Data and Applications Center (SEDAC) ?data_center=SEDAC
• [ ] UNAVCO ?data_center=UNAVCO
• [ ] Land Processes Distributed Active Archive Center ?data_center=LP DAAC
• [ ] Global Biodiversity Information Facility ?data_center=GBIF
• [ ] Ocean Biology Data Active Archive Center (OB.DAAC) ?data_center=NASA/GSFC/SED/ESD/GCDC/OB.DAAC
• [ ] Physical Oceanography Distributed Active Archive Center ?data_center=NASA/JPL/PODAAC
• [ ] National Center for Atmospheric Research ?data_center=UCAR/NCAR/CISL/RDA
• [ ] Level-1 and Atmosphere Archive & Distribution System (LAADS) Distributed Active Archive Center (DAAC) ?data_center=NASA/GSFC/SED/ESD/HBSL/BISB/LAADS
• [ ] NASA ESDIS Project ?data_center=NASA/GSFC/EOS/ESDIS
• [ ] INTERMAGNET ?data_center=INTERMAGNET

[yemoski]

You can get all of the data center urls by building a big query with each of their parameters as listed, like this: https://cmr.earthdata.nasa.gov/search/collections?data_center=AU/AADC&data_center=WGMS

[yemoski]

OK, our next step is to make a query that grabs all the urls for the datasets from the data centers in the first list, and then turn it into a sitemap that lets Gleaner crawl it.

[yemoski]

https://cmr.earthdata.nasa.gov/search/keywords/providers?pretty=true lists all the providers. so step 1 is to cross-reference them with our list of polar repositories so that we can just get datasets from the ones we want. @oluwayemisi4 is working on this.

[yemoski]

OK, here's how to do this: https://cmr.earthdata.nasa.gov/search/collections?provider=AU_AADC returns something that's like a sitemap for AADC datasets. We can request that, turn it into an actual sitemap (like we did for BAS), and crawl it.

What else is in the GCMD that we want?

[yemoski]

This is the same situation as the AMD - there's no json-ld in here, but there is API access, so we're going to have to figure something out.