Better support for dev takeover and testing new ManifestSync yamls

[jlewi]

With the addition of the Repo resource #59 we now support building a whole bunch of resources (Images, Manifests) that comprise an application.

One of the friction points right now is doing a takeover. Right now to do a dev takeover the usual pattern is to create a second manifestsync.yaml. That's kind of annoying.

An alternative would be to use kustomize functions to replace all the git references in a bunch of resources to point to a branch or even a local path.

One thing this doesn't solve is how do we selectively enable rebuilding certain images but not others?

[jlewi]

One might expect the following to work.

1. Specify a RepoConfig pointing at a branch; e.g.

   apiVersion: hydros.sailplane.ai/v1alpha1
   kind: RepoConfig
   metadata:
   name: repo
   namespace: sailplaneai
   spec:
   repo: https://github.com/sailplaneai/code.git?ref=jlewi/cicd
   ..
   globs:
   - "**/*.yaml"
   selectors:
   - matchLabels:        env: dev

2. Apply it

hydros apply -f repoconfig

This doesn't work though because RepoConfig won't override the repositories referenced in ManifestSync. So they could end up pointing at Main which means the images that do get built will have the commit of the branch but the ManifestSync will look for the commit corresponding to main so it won't find the images and hydration won't succeed.

This is a perfect use case for a kustomize function. RepoConfig uses findresources https://github.com/jlewi/hydros/blob/071b01b535b3f6c2ca718fb0f9edf2721b5211bb/pkg/gitops/repocontroller.go#L157

So we could apply any transformations in memory.

[jlewi]

75 works pretty well for deploying a bunch of microservices.

The current pain point would be that with #75 you have to rebuild all images on every commit which would be very expensive. I think a better approach would be

• Create a unique but mutable tag for the "deployment"
• Build and tag all images with that mutable tag
• When hydrating pin the image to the current version with that mutable tag
• Let the user selectively rebuild and retag a given image
• When manifests get resync'd if the mutable tag has moved the manifests should be rehydrated
  • The image pinning already happens with tag strategy mutable. I think the part that might need to change is that when deciding whether a manifest needs to be resync'd we need to take into account whether the mutable tag has moved to a new image. Rather than just looking at the sourceCommit.

[jlewi]

A related problem is the developer flow when defining a new manifestsync.yaml for a new service. How can you use hydros to test that everything is working and deploy it through gitops? This is effectively a dev-takeover but

1. You don't want to have to author a separate ManifestSync.yaml from the one that will be auto-deployed
2. You don't want to have to rebuild the image on every reconcile; so you'd like to use a mutable tag

We can't use a "takeover" because that will try to push any commits on your local branch to the sourceRepo listed in the manifestsync.yaml. However, the sourceRepo will be pointing at the main branch not the dev takeover branch.