Argocd image updater stuck on specific tag - not updating .argocd-source-myapp.yaml unless tag is deleted from docker registry

[dren-sokoli]

Checklist:

• [ x ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ x ] I've included steps to reproduce the bug.
• [ x ] I've pasted the output of argocd version.

Describe the bug

I am using argocd to manage my PRODUCTION Kubernetes cluster, and I have configured the argocd image updater to update the image with the latest tag from Docker Hub using the git write-back method which has worked well for the past couple of months. However, I have encountered two occurrences where the image updater does not update the image to the latest tag. Even when I manually update (edit the .argocd-source-myapp.yaml) the tag to a more recent version, it reverts back to the specific tag, which is an older version. I need help understanding why this is happening and how to resolve it.

argocd image updater logs:

time="2023-08-04T14:39:00Z" level=info msg="Successfully updated image 'docker.io/<user>/<image_name>:95083' to 'docker.io/<user>/<image_name>:95057', but pending spec update (dry run=false)" alias= application=<application_name> image_name=<user>/<image_name> image_tag=95083 registry=docker.io

As you can see, the argocd image updater is updating the image to an older version - from 95083 to 95057

To Reproduce

Configure argocd image updater with the following settings:

argocd-image-updater.argoproj.io/image-list: docker.io/<user>/<image-name>
argocd-image-updater.argoproj.io/write-back-method: git
argocd-image-updater.argoproj.io/git-branch: main
argocd-image-updater.argoproj.io/update-strategy: latest

Push a new tag to Docker Hub for the specified image. Observe that the argocd image updater does not update the image in the .argocd-source-codex-api-content-development.yaml file.

Expected behavior

The argocd image updater should update the image in the specified file to the latest tag from Docker Hub.

Version

v2.7.1+5e54351.dirty

Logs

time="2023-08-04T14:39:00Z" level=info msg="Successfully updated image 'docker.io/<user>/<image_name>:95083' to 'docker.io/<user>/<image_name>:95057', but pending spec update (dry run=false)" alias= application=<application_name> image_name=<user>/<image_name> image_tag=95083 registry=docker.io

[ks-yim]

Would you please compare the build timestamp of those 2 images? The documentation about the latest update strategy says:

It is important to understand, that this strategy will consider the build date of the image, and not the date of when the image was tagged or pushed to the registry.

And I've also experienced a similar issue when we were using an immutable build tool like jib. (though I don't think you are using it if I look at the logs you've posted)

[dren-sokoli]

Hi @ks-yim , I build the images using Azure DevOps build pipelines which uses only one VM with 3 agents in it. Apparently (somehow), images that have been build at a more recent time, sometimes have a timestamp that is older than images that have been built earlier. So the issue seems to be with the image build timestamp. Has anyone ever reported a similair issue (I checked but couldn't find any). If so, do you have any idea how I could overcome this? Each new image contains a larger (numeber) build tag than the previous image. I thought the name update-strategy (that uses lexical sort) would come in handy, but this doesn't work well with numbers, only letters.

[ks-yim]

@dren-sokoli What tool do you use to build an image in Azure DevOps pipeline?

name strategy will work just fine if your image tags' number part is correctly padded, I guess. If this is not the case, maybe you should change your tag schema a bit so that the lexicographic order becomes the same as build-time order (e.g. using calver).

[dren-sokoli]

@ks-yim We use the official Dockerhub build and push task for our docker images. We use the $(Build.BuildId) variable as the image tag and changing the tag schema is unfortunately not an option. As per your suggestion of using calver, I can see in the documentation that it uses versioning convention based on the project's release calendar, which is exactly where the issue lies, because the build timestamps are being created in a non-conventional way. Thanks for the suggestion though! I will try to find a different solution for now, but if you find anything related to this issue I would really appreciate further input!

[driv]

Docker hub is showing me the tag times. How can I see the build times?

[stefan-korchahin]

I have exactly the same bug! Have you any workarounds? Argo not even pushing into .argocd-source file, just one app isn't working, it happens unexpectedly, a few weeks ago it works normally. I'm using AWS ECR, all access creds are ok, I even not see any errors during this problem.

      annotations:
        argocd-image-updater.argoproj.io/image-list: repo=12321421.dkr.ecr.eu-west-3.amazonaws.com/frontend-nginx-dev
        argocd-image-updater.argoproj.io/repo.update-strategy: latest
        argocd-image-updater.argoproj.io/repo.force-update: "true"
        argocd-image-updater.argoproj.io/write-back-method: git:secret:argo-cd/dev-apps
        argocd-image-updater.argoproj.io/git-branch: master
        argocd-image-updater.argoproj.io/repo.helm.image-name: image.repository
        argocd-image-updater.argoproj.io/repo.helm.image-tag: image.tag
        argocd-image-updater.argoproj.io/repo.pull-secret: pullsecret:argo-cd/regcred

[stefan-korchahin]

The issue was an absence linux/arm64/v8 platform in argocd-image-updater.argoproj.io/imagealias.platforms: linux/amd64,linux/arm64,linux/arm64/v8