Image tag updates

[derrickburns]

IOne of the nice features of Weave Flux is the ability to monitor a docker image repo for the publication of new images that match a given pattern, such as a regex or semver.

Then, Flux updates the image tag in the Git repo that stores manifests equivalent to the Argo argoproj.io/v1alpha1/Application manifest. This is very elegant and useful.

This can be achieved because Weave Flux externalizes the storage of the Application manifest (they call the HelmReleases) into GitHub.

Is there an equivalent with Argo-CD? Can the applications be stored in GitHub and synced? If so, is also an equivalent way to update them automatically when new Docker images are published.

I am using Weave Flux, but prefer the GUI of Argo.

[reegnz]

I would welcome such a great component that complements the current ArgoCD functionality! Even if this doesn't find it's way into argocd itself, such a component could be fully self-contained, eg. docker registry monitoring capability, git commit capability to patch manifests and a rule engine to define the rules on when to patch. I'd say this functionality could be split out from flux into it's own component, as it's already implemented there, and is totally self-contained (not mixed with the deploy logic). I'm not that proficient yetin golang to do it myself though...

[alexec]

Yes. Applications can be stored in Github. We call this "declarative set-up".

https://argoproj.github.io/argo-cd/operator-manual/declarative-setup/

[alexec]

However, if you want to automatically update them, that's kind of out of the purvue of Argo CD (it never writes to Git). However, amongst other solutions, you could create a nightly job in Jenkins that does this. There maybe other solutions.

[alexmt]

I think this can be implemented as a separate component which watches the docker registry and updates git repo. The component don't have to be embedded into Argo CD and could be used with or without Argo CD.

This would be great contribution to https://github.com/argoproj-labs

[Alwinius]

For my Bachelor's Thesis about GitOps and ArgoCD I forked keel.sh to implement this feature. You can check it out here: https://github.com/Alwinius/keel The basics (and advanced keel features) should be working, but I still consider it alpha state since I only tested it with one deployment repository. Feel free to submit bug reports if you encounter something broken.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[reegnz]

I've looked around the https://github.com/argoproj/argoproj-deployments repo, and found some interesting things that are very much relevant to this issue: https://github.com/argoproj/argoproj-deployments/blob/master/argocd/overlays/production/argo-cd-cm.yaml#L7 https://github.com/argoproj/argoproj-deployments/blob/master/argocd/overlays/production/argocd-repo-server-deploy.yaml#L20

So it seems they are already experimenting with a flux plugin: https://github.com/alexmt/argocd-flux. So that's just a hint on where to start looking, I will be trying that out once I get around to it. :)

[reegnz]

OK, that just seems to implement the deploy part of flux, not the image updater part. :(

[alexec]

This is an eschewed feature.

Why?

One defining principle of Argo CD is that Git is the source-of-truth. Unfortunately, this is not always the case, common examples are where an app has a dependency that can change with any changes in Git.

• Helm chart dependencies, that are not pinned
• Kustomize remote bases, that are not pinned
• Docker images, that are not pinned ;)

Argo CD cannot monitor these for updates, and they are not audited by. Instead, you should pin your external dependencies and have an external process that updates Git when the external dependency updates.

What Can I Do?

If you build the image yourself, then your CI job could:

• Update the manifests, e.g kustomize edit set image $IMAGE .
• Commit the changes, e.g. git commit -am "Update image to $IMAGE".
• For manually synced apps, trigger the sync argocd app sync $APPNAME.

For images build elsewhere, and you do cannot pin the image, you could simple ask Argo CD to sync on a cronjob.

[Morriz]

We have a mono repo that sets up namespaces with core services for teams to run their apps in, but we don’t allow them access to our repo. They don’t even have kube api access and can only push new images that have to be picked up somehow automatically so that the new tag can be committed by the operator into the monorepo.

No dependencies which can’t be handled through semver tagging, and we have the possibility to let the teams manually set the semver match string through a ui (we instruct them to do controlled breaking-change updates by upping the semver match, but we want to offer them automated deployments for patches).

Very predictable workflow, very much in need of this functionality. Don’t you agree @alexec that such would fit perfectly in Argo, as it did in flux?

[timothyjlaurent]

We store all of our deployments as ArgoCD Applications. Upon a successful build we'll trigger a downstream Jenkins pipeline residing in the gitops repo with the Image tag, the Repo url, and the branch as parameters. We keep a map of repo urls -> to branch patterns -> to yaml paths to update. Each repo also has a docker image and command template with parameters slots for the image tag and file name.

So if the CI script cares about a giturl, branch pattern, then it will update the manifest files in the application.yaml with the given command template, the commit the result to the repo. Argo Will then initiate the update for the new tag.

[michaelgeorgeattard]

Do you guys have a timeline for this feature to be merged to ArgoCD?

[metaversed]

@alexec if were to use Azure DevOps as the CI tool any inputs on how we can handle this if feature is not provided by ARGO CD on purpose ?

[jannfis]

Just a quick heads-up - Argoproj Labs is currently incubating https://github.com/argoproj-labs/argocd-image-updater

It is neither production ready nor feature complete yet, but we'd appreciate early adopters, ideas and contributions.

[StarpTech]

This is an eschewed feature.

Why?

One defining principle of Argo CD is that Git is the source-of-truth. Unfortunately, this is not always the case, common examples are where an app has a dependency that can change with any changes in Git.

• Helm chart dependencies, that are not pinned
• Kustomize remote bases, that are not pinned
• Docker images, that are not pinned ;)

Argo CD cannot monitor these for updates, and they are not audited by. Instead, you should pin your external dependencies and have an external process that updates Git when the external dependency updates.

What Can I Do?

If you build the image yourself, then your CI job could:

• Update the manifests, e.g kustomize edit set image $IMAGE .
• Commit the changes, e.g. git commit -am "Update image to $IMAGE".
• For manually synced apps, trigger the sync argocd app sync $APPNAME.

For images build elsewhere, and you do cannot pin the image, you could simple ask Argo CD to sync on a cronjob.

This will ends up in a scenario like GitHub and Bots where third-party apps can commit to your repo 😄 This has a great potential for merge conflicts.

We currently evaluating ArgoCD and we faced a very similar use case as mentioned in https://github.com/argoproj/argo-cd/issues/1648#issuecomment-581612729. We would like to roll out patches automatically (semver lock) for preview deployments. In that way, we don't need to maintain the versions in the helm values because the communication of updates is hard if you work on many microservices with different teams.

The needed functionality could also be developed as a one-time operation before or in the CI.

1. argocd sync semver: Updates all deployment images with regards to semver constraints.
2. Commit or sync manually with argocd

[maxbrunet]

We have started experimenting with the Flux2 services. Since its features have been broken down into different services, the ones responsible for watching the image registry and updating the manifests in Git can be deployed along side Argo CD:

• https://github.com/fluxcd/image-automation-controller
• https://github.com/fluxcd/image-reflector-controller
• https://github.com/fluxcd/source-controller

Documentation: https://toolkit.fluxcd.io/guides/image-update/

A Kustomize deployment can be started like that:

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/fluxcd/image-automation-controller/config/crd?ref=v0.3.1
- github.com/fluxcd/image-automation-controller/config/manager?ref=v0.3.1
- github.com/fluxcd/image-automation-controller/config/rbac?ref=v0.3.1
- github.com/fluxcd/image-reflector-controller/config/crd?ref=v0.3.0
- github.com/fluxcd/image-reflector-controller/config/manager?ref=v0.3.0
- github.com/fluxcd/image-reflector-controller/config/rbac?ref=v0.3.0
- github.com/fluxcd/source-controller/config/crd?ref=v0.6.3
- github.com/fluxcd/source-controller/config/manager?ref=v0.6.3
- github.com/fluxcd/source-controller/config/rbac?ref=v0.6.3

Note that these services are still at an early stage of development.

Argo CD + Flux = ❤️

[Anishmourya]

I am using Keel.sh with the deployments option, Since ArgoCD support helm script to installation but doesn't use Helm way to install so keel doesn't watch ArgoCD applications but it can watch Kubernetes deployments which ArgoCD deploy the project as deployment. You can use the deploymentSet option to update the docker registry image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
  labels:
    keel.sh/policy: force
    keel.sh/match-tag: "true"
    keel.sh/trigger: poll
  annotations:
    keel.sh/pollSchedule: "@every 1m

I think its the easiest way to integrate keel with ArgoCD 😄 ❤️

[jessesuen]

Closing this, since the https://github.com/argoproj-labs/argocd-image-updater project was created to address this use case in a manner that allowed us to do it outside of core Argo CD.

[vikas027]

I am using Keel.sh with the deployments option, Since ArgoCD support helm script to installation but doesn't use Helm way to install so keel doesn't watch ArgoCD applications but it can watch Kubernetes deployments which ArgoCD deploy the project as deployment. You can use the deploymentSet option to update the docker registry image.

Hey @Anishmourya , Does keel also updates the git repository as well with the image tag as well?

[Anishmourya]

nahhh it just update the Kubernetes deployment application image, not the CloudOps tag. @vikas027

[rodrigopztpedro900]

Is posible use keel with argocd with kind rollout, from argo rollout?

[joebowbeer]

@rodrigopztpedro900 I think it is possible. Keel watches for changes in the image registry and then updates the spec.template, which in turn signals Argo Rollouts. I like this separation of concerns.

[rodrigopztpedro900]

I try to add the tags to the manifest of the kind rollout and it doesn't do anything keel. I think that when it detects that it is kind rollout and just ignores it I can move some keel or argo configuration to make this happens?

[TommyBeaton]

Keel looks for Deployments, Daemonsets and Statefulsets so it won't find the Rollout as it's deployed a Rollout instead of a Deployment :'(

[joebowbeer]

keel creates a new ReplicaSet and then scales down the old ReplicaSet, so it is effectively doing the rollout itself, but with less finesse and control than Argo Rollouts.

[TommyBeaton]

I've moved over to using Kobold instead which works with Rollouts + writes changes to Git so it's super nice to use (though I had a couple of issues setting it up). If anyone is looking for a good alternative to Image Updater I would recommend that :)

[tirelibirefe]

https://github.com/argoproj/argo-cd/issues/2532#issuecomment-1825496884

[joebowbeer]

Also see https://github.com/akuity/kargo which leverages ArgoCD's image updater