multiple image in one application, updater keeps looping the second image version between default and desired value

[mattdecorniquet]

Describe the bug Hi ! We have an issue where we use 2 images in a single helm application, configured image-updater to monitor and update both those images and this doesn't work reliably. Sometimes both images are updated, sometimes only the first in the list, and sometimes it updates both, then deletes the values for the second image in the .argocd-source file, then re-adds it, then redeletes it etc etc.

This is very similar and perhaps identical to #202, for which the fix still seems to be present in the code base. It's also very similar to #365 and #170.

Our annotations look like this:

    metadata:
      name: '{{`{{env}}`}}'
      annotations:
        argocd.argoproj.io/manifest-generate-paths: .:../../helm/ciam-app
        argocd-image-updater.argoproj.io/image-list: '{{`{{env}}`}}=gcr.io/xxxx/app, {{`{{env}}`}}-liquibase=gcr.io/xxxx/liquibase'
        argocd-image-updater.argoproj.io/{{`{{env}}`}}.helm.image-name: image.name
        argocd-image-updater.argoproj.io/{{`{{env}}`}}.helm.image-tag: image.tag
        argocd-image-updater.argoproj.io/{{`{{env}}`}}-liquibase.helm.image-name: imageLiquibase.name
        argocd-image-updater.argoproj.io/{{`{{env}}`}}-liquibase.helm.image-tag: imageLiquibase.tag
        argocd-image-updater.argoproj.io/git-branch: main
        argocd-image-updater.argoproj.io/write-back-method: git:secret:argocd/argocd-repo-creds-ssh-creds
        argocd-image-updater.argoproj.io/{{`{{env}}`}}.update-strategy: latest
        argocd-image-updater.argoproj.io/{{`{{env}}`}}.allow-tags: '{{`{{regexp}}`}}'
        argocd-image-updater.argoproj.io/{{`{{env}}`}}-liquibase.update-strategy: latest
        argocd-image-updater.argoproj.io/{{`{{env}}`}}-liquibase.allow-tags: '{{`{{regexp}}`}}'

Our helm chart specifies this:

    name: gcr.io/xxxx/app
    pullPolicy: IfNotPresent
    # Overrides the image tag whose default is the chart appVersion.
    tag: v1.0.1
imageLiquibase:
    name: gcr.io/xxxx/liquibase
    pullPolicy: IfNotPresent
    # Overrides the image tag whose default is the chart appVersion.
    tag: v1.0.1

In our build process, the liquibase image finishes first and is then pushed to the registry, and about 15mn later the app image is pushed. So we first see the following behaviour:

and then

So far so good but then, after a period of time, sometimes the next update cycle, or like in this instance more than an hour later, we get the following cycle starting:

And this keeps going on and on until we manually override the chart with the desired version (v1.0.1 being a dummy value).

To Reproduce I'm going to copy #365 here because that' really all we've done:

• Track multiple images in argocd-image-updater.argoproj.io/image-list annotation
• Bump version up in all tracked images at once or consequently
• Check commit on github

Expected behavior We would expect image-updater to only update each image once when it detects the latest valid tag and to not flap between forgetting and recognizing the second image in the list.

Additional context To note, we have 2 different environments with the same config (aside from the tag matching regexp), and the other one is working fine, albeit that looking at the logs it keeps saying "updating liquibase from v1.0.1 to v2.xxxx" even when it has already commited the change, and those log lines don't correlate to any commit being pushed. Both env had the issue on 0.12.2 before, at the time the config only had

       argocd-image-updater.argoproj.io/image-list: '{{`{{env}}`}}=gcr.io/xxxx/app, {{`{{env}}`}}-liquibase=gcr.io/xxxx/liquibase'
        argocd-image-updater.argoproj.io/{{`{{env}}`}}-liquibase.helm.image-name: imageLiquibase.name
        argocd-image-updater.argoproj.io/{{`{{env}}`}}-liquibase.helm.image-tag: imageLiquibase.tag

and we thought we fixed it by adding

        argocd-image-updater.argoproj.io/{{`{{env}}`}}.helm.image-name: image.name
        argocd-image-updater.argoproj.io/{{`{{env}}`}}.helm.image-tag: image.tag

Version 0.12.2

Logs

time="2023-05-02T13:28:15Z" level=info msg="Starting image update cycle, considering 15 annotated application(s) for update"
time="2023-05-02T13:28:16Z" level=info msg="Setting new image to gcr.io/xxxx/app:v2.92.0-RC1" alias=sbel1-ciam application=sbel1-ciam image_name=xxxx/app image_tag=v2.91.2 registry=gcr.io
time="2023-05-02T13:28:16Z" level=info msg="Successfully updated image 'gcr.io/xxxx/app:v2.91.2' to 'gcr.io/xxxx/app:v2.92.0-RC1', but pending spec update (dry run=false)" alias=sbel1-ciam application=sbel1-ciam image_name=xxxx/app image_tag=v2.91.2 registry=gcr.io
time="2023-05-02T13:28:16Z" level=info msg="Committing 1 parameter update(s) for application sbel1-ciam" application=sbel1-ciam
time="2023-05-02T13:28:16Z" level=info msg="Initializing git@github.com:ReachFive/ciam-helm.git to /tmp/git-sbel1-ciam3108288644"
time="2023-05-02T13:28:16Z" level=info msg="rm -rf /tmp/git-sbel1-ciam3108288644" dir= execID=c4863
time="2023-05-02T13:28:16Z" level=info msg=Trace args="[rm -rf /tmp/git-sbel1-ciam3108288644]" dir= operation_name="exec rm" time_ms=0.928158
time="2023-05-02T13:28:16Z" level=warning msg="temporarily disabling strict host key checking (i.e. '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'), please don't use in production"
time="2023-05-02T13:28:16Z" level=info msg="git fetch origin --tags --force" dir=/tmp/git-sbel1-ciam3108288644 execID=10cc5
time="2023-05-02T13:28:19Z" level=info msg=Trace args="[git fetch origin --tags --force]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=3060.1171640000002
time="2023-05-02T13:28:19Z" level=info msg="git config user.name argocd-image-updater" dir=/tmp/git-sbel1-ciam3108288644 execID=a2864
time="2023-05-02T13:28:19Z" level=info msg=Trace args="[git config user.name argocd-image-updater]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=1.544723
time="2023-05-02T13:28:19Z" level=info msg="git config user.email noreply@argoproj.io" dir=/tmp/git-sbel1-ciam3108288644 execID=6a26c
time="2023-05-02T13:28:19Z" level=info msg=Trace args="[git config user.email noreply@argoproj.io]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=1.618967
time="2023-05-02T13:28:19Z" level=info msg="git checkout --force main" dir=/tmp/git-sbel1-ciam3108288644 execID=819c6
time="2023-05-02T13:28:19Z" level=info msg=Trace args="[git checkout --force main]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=47.498165
time="2023-05-02T13:28:19Z" level=info msg="git clean -fdx" dir=/tmp/git-sbel1-ciam3108288644 execID=c10c3
time="2023-05-02T13:28:19Z" level=info msg=Trace args="[git clean -fdx]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=3.217558
time="2023-05-02T13:28:19Z" level=info msg="git commit -a -F /tmp/image-updater-commit-msg4263898099" dir=/tmp/git-sbel1-ciam3108288644 execID=75020
time="2023-05-02T13:28:19Z" level=info msg=Trace args="[git commit -a -F /tmp/image-updater-commit-msg4263898099]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=20.135365
time="2023-05-02T13:28:19Z" level=warning msg="temporarily disabling strict host key checking (i.e. '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'), please don't use in production"
time="2023-05-02T13:28:19Z" level=info msg="git push origin main" dir=/tmp/git-sbel1-ciam3108288644 execID=92a0f
time="2023-05-02T13:28:20Z" level=info msg=Trace args="[git push origin main]" dir=/tmp/git-sbel1-ciam3108288644 operation_name="exec git" time_ms=1485.334605
time="2023-05-02T13:28:20Z" level=info msg="Successfully updated the live application spec" application=sbel1-ciam
time="2023-05-02T13:28:20Z" level=info msg="Processing results: applications=15 images_considered=14 images_skipped=9 images_updated=1 errors=0"

[stephaneetje]

We have the same bug, looks related to this issue