NASA-Openscapes / earthdata-cloud-cookbook

A tutorial book of workflows for research using NASA EarthData in the Cloud created by the NASA-Openscapes team
https://nasa-openscapes.github.io/earthdata-cloud-cookbook
Other
83 stars 30 forks source link

Containerized environment(s) for earthdata cloud cookbook #306

Closed cboettig closed 3 months ago

cboettig commented 4 months ago

Provides three GitHub Actions (https://github.com/eeholmes/earthdata-cloud-cookbook/actions):

  1. Docker Image CI: builds a Hub-compatible docker image (with support for RStudio and VSCode proxies)
  2. Quarto Publish: same as original, but building the cookbook inside the image (with _freeze cache)
  3. Quarto Test Build: Build the cookbook from scratch on the docker image to test; monthly, on push changes, or manual trigger

This is set up so that all actions pass, but this isn't all working perfectly yet. Specifically, I could not get all the cookbook recipes to re-build, even when running on the Openscapes hub. A few may need updates, while others look like they aren't designed to run outside of AWS us-west-2, maybe we need a better way to document that and run conditionally? For now, quarto is building without the --execute flag, effectively meaning that the .qmd / R notebooks get tested by Test Build, but no the the Jupyter ipynbs's.

There are also actually two Docker tags being built on this CI -- one provides a virtualenv-based environment, the other a conda-based one. The conda-based one is a bit larger (~ 13 GB), almost too large to be easily used in some contexts (e.g. codespaces small machines and some github actions runners don't have sufficient disk space). But these images could be streamlined or swapped out entirely, the main point of this PR is to illustrate that it's easy to adjust the underlying container and then re-use that container across multiple settings (e.g. laptop, gh-actions tests, codespaces, openscapes hub, etc).

jules32 commented 4 months ago

Hi @cboettig this is so awesome! 🚀 We can plan to merge this in Coworking on Thursday with the Mentors; excited to discuss together about it. Thank you!