Example pipeline for CESM POP low-resolution (1 degree)

[paigem]

Source Dataset

This is the ocean post-processed data of a low-resolution (1 degree ocean and atmosphere) Community Earth System Model (CESM) run: v5_rel04_BC5_ne30_g16. This is the low-resolution counterpart to the CESM run hybrid_v5_rel04_BC5_ne120_t12_pop62 with 0.1 degree ocean/0.25 degree atmosphere that is already available in the Pangeo Cloud Data Catalog here. Data is output as daily averages for a total of 166 model years. The data current sit on the Climate Data Gateway at NCAR.

• The website to download the data is here, but requires authentication to access (see below). A publicly accessible website that lists the variables can be found here.
• The file format is netCDF.
• There is one netCDF file per variable, with 14 variables total. Each of the 14 netCDF files are between 15GB and 20GB, which together sum to about 251GB.
• Source files can be accessed via wget or curl
  • Scripts to run wget or curl are provided after logging in from this page, selecting all of the listed files, and clicking "Download Options for Selections".
• Authentication is required to download the data. I was able to access using my UCAR CIT account, but it appears that there are three authentication options on this page, after clicking "Download Options":
  • use a UCAR CIT account
  • use an OpenId account
  • register for a guest account at Climate Data Gateway

Transformation / Alignment / Merging

The files should be combined into one dataset comprising all 14 variables, so it can be loaded in, e.g., as an Xarray Dataset.

Output Dataset

The files should be stored in the Zarr format.

[paigem]

Update on the authentication needed: it looks like the second two options above are actually the same (i.e. registering for an account at the Climate Data Gateway actually gives you an OpenID account. It appears that anyone is able to register for an OpenID account, as I just did in about 2 minutes (requires name, password, email).

[cisaacstern]

@paigem, apologies for the delay on this! To kick off our work, could I ask you to do the following?

1. Fork https://github.com/pangeo-forge/staged-recipes
2. Clone your fork to your local machine with:

   git clone https://github.com/paigem/staged-recipes.git

3. Make a PR branch on your fork:

   git checkout -b cesm-pop-lowres-1deg

4. In your PR branch, make a new directory for our recipe with a blank recipe.py:

   mkdir recipes/cesm-pop-lowres-1deg
   touch recipes/cesm-pop-lowres-1deg/recipe.py

5. Commit and push changes:

   git add -A
   git commit -m "first commit"
   git push origin cesm-pop-lowres-1deg

6. Follow the prompts at https://github.com/paigem/staged-recipes.git to open a PR against pangeo-forge/staged-recipes, and mention this Issue by number (#46) in the PR comment

Apologies if some/all of the above git workflow is obvious! I started writing and then thought, "you know, this is a good chance to write down some generalizable instructions," so I just went for it! :smile:

As a maintainer, I will then be able to push changes to your PR branch, and our collaboration on this will be in full swing! 🎸

[paigem]

Thanks @cisaacstern for such clear instructions! Specific git instructions are always appreciated. 🙂

I just put a PR in for steps 1-5 above, but need a bit of guidance with number 6, as discussed in the PR comment #56.