add promo mech that can update or sync multiple apps managed by a single appset

[tal-hason]

when configuring stage we have 3 type of connection points , subscription, promotionMechanisms and argoCDAppUpdates.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: develop
  namespace: gotham-kargo
  annotations:
    kargo.akuity.io/color: blue
spec:
  subscriptions:
    repos:
      images:
      - repoURL: quay.io/gotham/batman
        updateStrategy: Latest
      - repoURL: quay.io/gotham/joker
        updateStrategy: Latest
      - repoURL: quay.io/gotham/robin
        updateStrategy: Latest
      - repoURL: quay.io/gotham/riddler
        updateStrategy: Latest
      - repoURL: quay.io/gotham/gotham
        updateStrategy: Latest
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/gotham-demo/gotham-cd.git #https://gitea-http.gitea.svc/gotham/gotham-cd.git
      writeBranch: main # Writing to stage-specific branch prevents the formation of a loop
      helm:
        images:
        - image: quay.io/gotham/batman
          valuesFilePath: Application/gotham-demo/develop/values-batman.yaml
          key: image.tag # Update this key to reference the new image version
          value: 'Tag'
        - image: quay.io/gotham/joker
          valuesFilePath: Application/gotham-demo/develop/values-joker.yaml
          key: image.tag # Update this key to reference the new image version
          value: 'Tag'
        - image: quay.io/gotham/robin
          valuesFilePath: Application/gotham-demo/develop/values-robin.yaml
          key: image.tag # Update this key to reference the new image version
          value: 'Tag'
        - image: quay.io/gotham/riddler
          valuesFilePath: Application/gotham-demo/develop/values-riddler.yaml
          key: image.tag # Update this key to reference the new image version
          value: 'Tag'
        - image: quay.io/gotham/gotham
          valuesFilePath: Application/gotham-demo/develop/values-gotham.yaml
          key: image.tag # Update this key to reference the new image version
          value: 'Tag'
    argoCDAppUpdates:
    - appName: gotham-batman-gh-develop
      appNamespace: openshift-gitops
    - appName: gotham-joker-gh-develop
      appNamespace: openshift-gitops
    - appName: gotham-robin-gh-develop
      appNamespace: openshift-gitops
    - appName: gotham-riddler-gh-develop
      appNamespace: openshift-gitops
    - appName: gotham-gotham-gh-develop
      appNamespace: openshift-gitops

as we can see there is a pattern here cause this set of applications is managed and deployed with an argo applicationSet so if we know that this specific stage is deployed in a pattern.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: develop
  namespace: gotham-kargo
  annotations:
    kargo.akuity.io/color: blue
spec:
  subscriptionSet:
    repos:
      images:
      - repoURL: quay.io/gotham/* # will query all the images in the registry and will populate a list of image names that will be later used to fill promotionMechanisms
        updateStrategy: Latest
  promotionMechanisms:
    gitRepoUpdates:
    - repoURL: https://github.com/gotham-demo/gotham-cd.git 
      writeBranch: main
      helm:
        images:
        - image: # This is not necessary anymore cause the controller will follow a pattern from the subscription
          valuesFilePath: Application/gotham-demo/develop/values-{{subscription.image.name}}.yaml #a wild card pattern that will be populated from the subscription
          key: image.tag # Update this key to reference the new image version
          value: 'Tag'
    argoCDAppUpdates:
    - appSetName: gotham-applicationSet #the name of the argo applicationSet
      appNamespace: openshift-gitops

Idea that generated in #879

[krancour]

promotionMechanisms:
  gitRepoUpdates:
  - repoURL: https://github.com/gotham-demo/gotham-cd.git 
    writeBranch: main
    helm:
      images:
      - image: # This is not necessary anymore cause the controller will follow a pattern from the subscription
        valuesFilePath: Application/gotham-demo/develop/values-{{subscription.image.name}}.yaml #a wild card pattern that will be populated from the subscription
        key: image.tag # Update this key to reference the new image version
        value: 'Tag'
  argoCDAppUpdates:
  - appSetName: gotham-applicationSet #the name of the argo applicationSet
    appNamespace: openshift-gitops

There are two separate things going on in this example that aren't intrinsically related to one another:

1. Update multiple values files
2. Force sync of multiple Applications by specifying an ApplicationSet.

The reason I say these are separate issues is that it's easy to imagine a scenario where a single Application deploys all your new images. In that case, you need the pattern part, but not the ApplicationSet part.

The opposite it also true. It's easy to imagine a scenario where you update only one values file, but you need to sync multiple Applications. In that case, you need the ApplicationSet part, but not the pattern part.

So I want to focus this issue on a single topic. I think #879, which was initially framed as "wildcard" support is driving in the direction of the pattern support you're talking about here, so let's handle pattern stuff in that thread and keep this thread focused on just the ApplicationSet support part.

[krancour]

@tal-hason also, not to detract from your valuable suggestion and also not to get us too sidetracked, but for your own demo / use case you're working on, you evidently want to promote to multiple Applications as a set. If that's your intention, it's functionally equivalent to using just one App that deploys all your charts/images. i.e. There's an option that immediately gets you part of what you're requesting.

[tal-hason]

Yeah, I know about helm sub-charts(umbrella), the issue with that pattern is that using it to deploys multiple deployments/components in a single argo application, which from my experience makes argo work hard on the template and causes stress on the controller. and overload the argoCD UI, so I prefer using a single well-customized template and multiple values to deploy multiple argo applications. it just works much better. ( i.e. the use case for the bookkeeper to remove load from the repo controller). and manage those with projects to aggregate them into a single type.

[krancour]

Your preference and concerns are valid. I just wanted to be sure you knew you had the option if you needed it. 😄

[tal-hason]

@krancour ,I think there should be a diff check between freights so only update the changed images/tags, it is not necessary to update all the YAMLs with tags that haven't been changed.

I noticed that Kargo updated the unchanged tag when moving between freights. ( new issue!? 🤔)

[krancour]

I'm not sure I understand...

[tal-hason]

@krancour , if my stage has 2-3 image subscriptions, then after 1 new image tag is pushed it will get a new freight ID. but this freight ID have all 2-3 image, even if just 1 has been changed, and in the commit it will update all 2-3 tag even those that hasn't changed, hope this is clear?!

[krancour]

@tal-hason

if my stage has 2-3 image subscriptions, then after 1 new image tag is pushed it will get a new freight ID.

I follow so far.

but this freight ID have all 2-3 image, even if just 1 has been changed

It's like this by design, and admittedly people are having a hard time wrapping their heads around this. I believe we have a series of blog posts in the works to explain this better.

A piece of freight isn't a change. It's a meta-artifact, if you will, composed of child artifacts. If any one thing changes, you have a new set.

and in the commit it will update all 2-3 tag even those that hasn't changed

It will try to update all of them, yes.

Kustomize provides the easiest scenario to illustrate. At each stage, kustomize edit set image will run for each image. For some of those images, it may result in no diff, and for others it may. If you subscribe to three images and only one is new WRT to a given stage, that commit that effects promotion to that stage will be just a one line diff.

No matter what, in the end, the stage reflects everything that was in the freight...

So I don't see what the problem is, exactly.

I'm missing something.

[tal-hason]

No Problem, just over-committing unnecessary change. i am trying to continue to play with it.

@krancour, so I have created a stage for each image to a branch.

it looks like when having multiple stages with image subscription the images section on the right side does not update correctly, let me know if to open an issue/bug on that?!

[krancour]

No Problem, just over-committing unnecessary change.

In what I describe, there is no unnecessary change. Now that I see what you're trying, I have a better sense of why you may be seeing unnecessary changes.

"Merging" parallel pipelines as you have done here isn't supported. I'm not saying it wouldn't ever be or that it shouldn't be, (nor am I saying the opposite) but what I am saying is it doesn't work the way you are imagining it does.

Here, you have five separate stages subscribed to five different image repos. The freight moving through batman-image, robin-image, etc. are different -- Not just different tags or versions. They're entirely different things.

When you promote from any one of those to ready-to-deploy, you're promoting a "box" with just one thing in it -- not all five things.

If this is working for you at all, it's a stroke of luck.

In short... don't do this. If you want to do it, open a separate issue for it, please.

[krancour]

Also, if I may ask, please try to keep each issues focused on just one topic. Things get too confusing otherwise. I'm not sure this is related to the idea of a promotion mechanisms that syncs multiple Apps belonging to one AppSet. If it's somehow related and I am misunderstanding, I apologize and would welcome clarification.

[tal-hason]

No Problem, just over-committing unnecessary change.

In what I describe, there is no unnecessary change. Now that I see what you're trying, I have a better sense of why you may be seeing unnecessary changes.

"Merging" parallel pipelines as you have done here isn't supported. I'm not saying it wouldn't ever be or that it shouldn't be, (nor am I saying the opposite) but what I am saying is it doesn't work the way you are imagining it does.

Here, you have five separate stages subscribed to five different image repos. The freight moving through batman-image, robin-image, etc. are different -- Not just different tags or versions. They're entirely different things.

When you promote from any one of those to ready-to-deploy, you're promoting a "box" with just one thing in it -- not all five things.

If this is working for you at all, it's a stroke of luck.

In short... don't do this. If you want to do it, open a separate issue for it, please.

I am just testing things, to see what works and what kind of interaction I can create from and to Kargo with other tools(tekton GitHub actions, etc.), I want to find a way to create disposable deployment using Kargo, so I am playing around

before the stages looked like this:

that's why I asked about the diff in tags in freights.

and yes it is also a drift from the topic but it is also kinda of related, hint kargo-generator in argoCD applicationSet controller 🤔

to many ideas....

[krancour]

Yep... your second diagram is what we've built for. I don't think the other one is an invalid idea. We just haven't built for that scenario (yet?).

[tal-hason]

i am testing it right now, for now, it gives me the ability to promote just a specific image to develop I will keep playing

[github-actions[bot]]

This issue has been automatically marked as stale because it had no activity for 90 days. It will be closed if no activity occurs in the next 30 days but can be reopened if it becomes relevant again.