search

pangeo-forge / user-stories

User stories to guide PF development
1 stars 0 forks source link

Don't rerun unchanged recipes #3

Open cisaacstern opened 2 years ago

cisaacstern commented 2 years ago

User Profile

As a recipe maintainer

User Action

I want to push commits to the default branch of my feedstock repository, and have the resulting production deployment only rerun new recipes or those that have changed, and not rerun unchanged recipes

User Goal

So that I can add or update certain recipes in my feedstock without rerunning all of them.

Acceptance Criteria

A mechanism to check the hash of all recipes at deployment time, and skip re-running if the hash matches the hash for the same recipe in the last production deployment

Linked Issues

In the order in which they should be merged:

  1. https://github.com/pangeo-forge/pangeo-forge-recipes/pull/349
  2. https://github.com/pangeo-forge/pangeo-forge-orchestrator/pull/63
  3. https://github.com/pangeo-forge/registrar/pull/36
cisaacstern commented 2 years ago

Just noting that the main blocker to implementing this feature a new of release pangeo-forge-recipes (which will include https://github.com/pangeo-forge/pangeo-forge-recipes/pull/349).

@sharkinsspatial, let's sync next week to determine if it's we want to just manually build bakery images for this upcoming release, or if we'd like to use this opportunity to start working on some of the items in https://github.com/pangeo-forge/user-stories/issues/9.

cisaacstern commented 2 years ago

With https://github.com/pangeo-forge/pangeo-forge-recipes/pull/359, we arguably no longer need https://github.com/pangeo-forge/pangeo-forge-orchestrator/pull/63, which would duplicate this information into the database/API. This would slightly reduce the number of lines required to retrieve this information, but at the cost of additional complexity in the database (a PR/migration would be required) and, perhaps a greater cost, the risk of the two storage locations (dataset and database) falling out of sync.