rabernat
commented
3 years ago Good question.
I imagine using prefect "labels" for this: https://docs.prefect.io/orchestration/agents/overview.html#labels
The recipes will get labeled (e.g. bakery:ldeo-gcp-us-central1 or something like that). This will route recipes to bakeries. When we review and add new recipes, we can make sure that the bakery owner is okay with supporting that particular recipe.
However, this does require the bakery owner (e.g. AI for Earth) to trust the pangeo-forge github org and recipe review process sufficiently.
One question around the interaction between datasets and bakeries: will a bakery "owner" be able to opt in / out of specific datasets?
If AI for Earth wanted to run a bakery, would we be expected to process and host every dataset that comes along? I ask because a.) we have a focus on sustainability, and b.) we need some way to forecast budget.
I suppose the hypothetical NCAR bakery makes it clear that certain recipes would require certain bakeries.