Create report on pangeo-forge bakery vs. regular rechunking workflow

hytest-org / hytest

https://hytest-org.github.io/hytest/

22 stars 10 forks source link

Create report on pangeo-forge bakery vs. regular rechunking workflow #383

Open amsnyder opened 9 months ago

amsnyder commented 9 months ago

How does the workflow for each compare? (standardization, level of effort, format, etc.)
What cloud resources will we be using to deploy the bakery? (this can help us estimate cost)
What code/repos do we need to build this? How will we need to adapt/modify each of these?

amsnyder commented 8 months ago

pangeo-forge bakery has standardized template for use to fill in to rechunk a dataset

thodson-usgs commented 8 months ago

In principle, the pangeo-forge tooling seems very useful if we can jump the starting hurdle. I'm working on a recipe to process SSEBOP data for @kjdoore. The recipe was easy but I'm still working through issues

thodson-usgs commented 8 months ago

Still stuck, so I create a recipe PR to pangeo-forge. I'll follow up in their coordination meeting on Monday.

thodson-usgs commented 8 months ago

The SSEBOP recipe seems to be working now, thanks to some assistance from the pangeo-forge community.

All-in-all the recipes are fairly straightforward. Debugging Beam pipelines was challenging but I suspect that will get easier with time.

Pros: community of experts willing to help (esp if we contribute); recipes run a variety of "runners"; easier to manage large projects

Cons: Beam is harder to use than a notebook; Pangeo-forge documentation is incomplete; may create some technical lock-in or necessitate external support