"NOAA's Climate Data Records (CDRs) are robust, sustainable, and scientifically sound climate records that provide trustworthy information on how, where, and to what extent the land, oceans, atmosphere and ice sheets are changing. These datasets are thoroughly vetted time series measurements with the longevity, consistency, and continuity to assess and measure climate variability and change. NOAA CDRs are vetted using standards established by the National Research Council (NRC)."
License
"Open Data. There are no restrictions on the use of this data."
Data Format
NetCDF
Data Format (other)
No response
Access protocol
S3
Source File Organization
for each variable there is one file per day. each file is a single time step. the folder hierarchy is as follows
no transformations needed. simple wildcard will be used to extract the datetime from the file for the FilePattern
Target Format
Reference Filesystem (Kerchunk)
Comments
this is a collection of datasets with many different variables. variables are separated in their own directories. I'll check these off as I create recipes for them and submit a PR
[ ] Aerosol Optical Thickness
arn:aws:s3:::noaa-cdr-aerosol-optical-thickness-pds
aws s3 ls --no-sign-request s3://noaa-cdr-aerosol-optical-thickness-pds/
[ ] Cloud Properties ISCCP
arn:aws:s3:::noaa-cdr-cloud-properties-isccp-pds
aws s3 ls --no-sign-request s3://noaa-cdr-cloud-properties-isccp-pds/
this dataset has 4 distinct products in "isccp" and 3 in "isccp-basic", with "HXG" absent from the basic subdir.
from the algo doc, " The HGH Product provides the monthly average of the HGG Product at each of eight times-of-day UTC. The HGM Product is the average of the eight HGH Products for each month.". So HGM, HGH, and HGG seem important and are present in both isccp folders. HXG product seems like raw-er data so might be less of a priority to include.
[ ] Cloud Properties Polar Orbiter
arn:aws:s3:::noaa-cdr-cloud-properties-polar-orbiter-nasa-pds
aws s3 ls --no-sign-request s3://noaa-cdr-cloud-properties-polar-orbiter-nasa-pds/
[ ] Hydrological Properties
arn:aws:s3:::noaa-cdr-hydrological-properties-pds
aws s3 ls --no-sign-request s3://noaa-cdr-hydrological-properties-pds/
[ ] Ocean Heat Content
arn:aws:s3:::noaa-cdr-ocean-heat-content-pds
aws s3 ls --no-sign-request s3://noaa-cdr-ocean-heat-content-pds/
[ ] Ocean Heatflux
arn:aws:s3:::noaa-cdr-ocean-heatflux-pds
aws s3 ls --no-sign-request s3://noaa-cdr-ocean-heatflux-pds/
[ ] Ocean Nearsurface Atmos Profiles
arn:aws:s3:::noaa-cdr-ocean-nearsurface-atmos-profiles-pds
aws s3 ls --no-sign-request s3://noaa-cdr-ocean-nearsurface-atmos-profiles-pds/
[ ] Outgoing Longwave Radiation - Daily
arn:aws:s3:::noaa-cdr-outgoing-longwave-radiation-daily-pds
aws s3 ls --no-sign-request s3://noaa-cdr-outgoing-longwave-radiation-daily-pds/
[ ] Outgoing Longwave Radiation - Monthly
arn:aws:s3:::noaa-cdr-outgoing-longwave-radiation-monthly-pds
aws s3 ls --no-sign-request s3://noaa-cdr-outgoing-longwave-radiation-monthly-pds/
[ ] Ozone - ESRL
arn:aws:s3:::noaa-cdr-ozone-esrl-pds
aws s3 ls --no-sign-request s3://noaa-cdr-ozone-esrl-pds/
[ ] CMORPH Precip
arn:aws:s3:::noaa-cdr-precip-cmorph-pds
aws s3 ls --no-sign-request s3://noaa-cdr-precip-cmorph-pds/
[ ] GPCP Precip Daily there's an existing recipe https://github.com/pangeo-forge/staged-recipes/pull/150/files in Zarr format but we'd still like to build a kerchuk recipe for GPCP that references the raw hdf files stored at s3://noaa-cdr-precip-gpcp-daily-pds/
arn:aws:s3:::noaa-cdr-precip-gpcp-daily-pds
aws s3 ls --no-sign-request s3://noaa-cdr-precip-gpcp-daily-pds/
[ ] GPCP Precip Monthly
arn:aws:s3:::noaa-cdr-precip-gpcp-monthly-pds
aws s3 ls --no-sign-request s3://noaa-cdr-precip-gpcp-monthly-pds/
[ ] NEXRAD Precip
arn:aws:s3:::noaa-cdr-precip-nexrad-qpe-pds
aws s3 ls --no-sign-request s3://noaa-cdr-precip-nexrad-qpe-pds/
[ ] PERSIANN Precip
arn:aws:s3:::noaa-cdr-precip-persiann-pds
aws s3 ls --no-sign-request s3://noaa-cdr-precip-persiann-pds/
[ ] Solar Spectral Irradiance Daily
arn:aws:s3:::noaa-cdr-solar-spectral-irradiance-pds
aws s3 ls --no-sign-request s3://noaa-cdr-solar-spectral-irradiance-pds/
[ ] Total Solar Irradiance Daily
arn:aws:s3:::noaa-cdr-total-solar-irradiance-pds
aws s3 ls --no-sign-request s3://noaa-cdr-total-solar-irradiance-pds/
Dataset Name
NOAA Atmospheric Climate Data Records
Dataset URL
https://registry.opendata.aws/noaa-cdr-atmospheric/
Description
"NOAA's Climate Data Records (CDRs) are robust, sustainable, and scientifically sound climate records that provide trustworthy information on how, where, and to what extent the land, oceans, atmosphere and ice sheets are changing. These datasets are thoroughly vetted time series measurements with the longevity, consistency, and continuity to assess and measure climate variability and change. NOAA CDRs are vetted using standards established by the National Research Council (NRC)."
License
"Open Data. There are no restrictions on the use of this data."
Data Format
NetCDF
Data Format (other)
No response
Access protocol
S3
Source File Organization
for each variable there is one file per day. each file is a single time step. the folder hierarchy is as follows
data/ monthly/ daily/ 1982/ 1983/ .... documentation/
we are proposing just to deal with the daily products
Example URLs
example urls
s3://noaa-cdr-aerosol-optical-thickness-pds/data/daily/1982/AOT_AVHRR_v04r00_daily-avg_19820101_c20220505.nc
s3://noaa-cdr-aerosol-optical-thickness-pds/data/daily/1982/AOT_AVHRR_v04r00_daily-avg_19820102_c20220505.nc
Authorization
No response
Transformation / Processing
no transformations needed. simple wildcard will be used to extract the datetime from the file for the FilePattern
Target Format
Reference Filesystem (Kerchunk)
Comments
this is a collection of datasets with many different variables. variables are separated in their own directories. I'll check these off as I create recipes for them and submit a PR
arn:aws:s3:::noaa-cdr-aerosol-optical-thickness-pds aws s3 ls --no-sign-request s3://noaa-cdr-aerosol-optical-thickness-pds/
arn:aws:s3:::noaa-cdr-cloud-properties-isccp-pds aws s3 ls --no-sign-request s3://noaa-cdr-cloud-properties-isccp-pds/
this dataset has 4 distinct products in "isccp" and 3 in "isccp-basic", with "HXG" absent from the basic subdir.
from the algo doc, " The HGH Product provides the monthly average of the HGG Product at each of eight times-of-day UTC. The HGM Product is the average of the eight HGH Products for each month.". So HGM, HGH, and HGG seem important and are present in both isccp folders. HXG product seems like raw-er data so might be less of a priority to include.
arn:aws:s3:::noaa-cdr-cloud-properties-polar-orbiter-nasa-pds aws s3 ls --no-sign-request s3://noaa-cdr-cloud-properties-polar-orbiter-nasa-pds/
arn:aws:s3:::noaa-cdr-hydrological-properties-pds aws s3 ls --no-sign-request s3://noaa-cdr-hydrological-properties-pds/
arn:aws:s3:::noaa-cdr-ocean-heat-content-pds aws s3 ls --no-sign-request s3://noaa-cdr-ocean-heat-content-pds/
arn:aws:s3:::noaa-cdr-ocean-heatflux-pds aws s3 ls --no-sign-request s3://noaa-cdr-ocean-heatflux-pds/
arn:aws:s3:::noaa-cdr-ocean-nearsurface-atmos-profiles-pds aws s3 ls --no-sign-request s3://noaa-cdr-ocean-nearsurface-atmos-profiles-pds/
arn:aws:s3:::noaa-cdr-outgoing-longwave-radiation-daily-pds aws s3 ls --no-sign-request s3://noaa-cdr-outgoing-longwave-radiation-daily-pds/
arn:aws:s3:::noaa-cdr-outgoing-longwave-radiation-monthly-pds aws s3 ls --no-sign-request s3://noaa-cdr-outgoing-longwave-radiation-monthly-pds/
arn:aws:s3:::noaa-cdr-ozone-esrl-pds aws s3 ls --no-sign-request s3://noaa-cdr-ozone-esrl-pds/
arn:aws:s3:::noaa-cdr-precip-cmorph-pds aws s3 ls --no-sign-request s3://noaa-cdr-precip-cmorph-pds/
arn:aws:s3:::noaa-cdr-precip-gpcp-daily-pds aws s3 ls --no-sign-request s3://noaa-cdr-precip-gpcp-daily-pds/
arn:aws:s3:::noaa-cdr-precip-gpcp-monthly-pds aws s3 ls --no-sign-request s3://noaa-cdr-precip-gpcp-monthly-pds/
arn:aws:s3:::noaa-cdr-precip-nexrad-qpe-pds aws s3 ls --no-sign-request s3://noaa-cdr-precip-nexrad-qpe-pds/
arn:aws:s3:::noaa-cdr-precip-persiann-pds aws s3 ls --no-sign-request s3://noaa-cdr-precip-persiann-pds/
arn:aws:s3:::noaa-cdr-solar-spectral-irradiance-pds aws s3 ls --no-sign-request s3://noaa-cdr-solar-spectral-irradiance-pds/
arn:aws:s3:::noaa-cdr-total-solar-irradiance-pds aws s3 ls --no-sign-request s3://noaa-cdr-total-solar-irradiance-pds/