search

pangeo-forge / pangeo-forge-orchestrator

Database API and GitHub App backend for Pangeo Forge Cloud.
https://api.pangeo-forge.org/docs
Apache License 2.0
4 stars 1 forks source link

Spec recipes 0.9.3 in requirements.txt #205

Closed cisaacstern closed 1 year ago

cisaacstern commented 1 year ago

@cmdupuis3 @rabernat, today's release of recipes will be available on the Pangeo Forge Cloud once this PR goes in. TL;DR I'm waiting on https://github.com/pangeo-data/pangeo-docker-images/pull/420. Long version below.

After I thought about it a bit more, I realized that my comment in https://github.com/pangeo-forge/pangeo-forge-recipes/issues/454#issuecomment-1359929931 was a bit of an oversimplification. In addition to releasing pangeo-forge-recipes, and the current change in this PR, we also need to:

This is all because we do not just need the new release in the FastAPI application here, but also for it to be available to Dataflow workers via a Docker image. (Eventual dynamic recipe environment creation will simplify this, but today this is where we are.) I'll make the change to the default image tag, convert this to Ready for review, and merge once the pangeo-docker-images PR goes through.

rabernat commented 1 year ago

Thanks for the summary Charles. What a pain! 😩 We really appreciate your efforts to push these updates through the stack.

I think this also concisely summarizes why "dynamic recipe environment creation" should be our top priority in terms of infrastructure.

cisaacstern commented 1 year ago

"dynamic recipe environment creation" should be our top priority in terms of infrastructure

Agreed! This is being tracked in https://github.com/pangeo-forge/pangeo-forge-runner/issues/27. As @yuvipanda has observed, making this pangeo-forge-runner's responsibility gives us a reproducibility boost: local calls to pangeo-forge-runner will build envs dynamically in the same way as the cloud calls will. Yuvi pointed out that pre-commit does something similar, and is a good design reference for us. This will be our next major focus after we can get all the beam-refactor stuff merged.