Image FOO seems not to be live in this application, skipping

[kallentrustle]

Describe the bug

I'm getting started using ArgoCD Image Updater (AIU). It's giving an error for one of my use cases where I deploy to GCP Cloud Run using ArgoCD. When a new docker image is ready for Cloud Run, I see this error message: "Image 'PROJECT_ID/MYREPO/hello-python' seems not to be live in this application, skipping" application=hello-thersitean-aiu-test".

As usual I'm hoping the problem is 'pebcak' and it's easy to spot .. Can anyone help?

In my example, hello-python is a docker image of a simple python app. hello-thersitean-aiu-test Application is a RunService that runs the hello-python image. (RunService is from Google Config Connector)

hello-thersitean-aiu-test launches with tag 0cf9946. It should upgrade to the latest tag b0d1559, but AIU isn't upgrading it. When the new tag lands on the repo, this is what I see in the AIU log:

time="2024-07-19T16:50:59Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2024-07-19T16:50:59Z" level=debug msg="Processing application hello-thersitean-aiu-test"
time="2024-07-19T16:50:59Z" level=debug msg="Image 'PROJECT_ID/MYREPO/hello-python' seems not to be live in this application, skipping" application=hello-thersitean-aiu-test
time="2024-07-19T16:50:59Z" level=info msg="Processing results: applications=1 images_considered=0 images_skipped=1 images_updated=0 errors=0"

For an application that is not a RunService, but a Deployment, AIU successfully updates it:

time="2024-07-16T18:15:41Z" level=info msg="Successfully updated image 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-python:0cf9946' to 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-python:b0d1559', but pending spec update (dry run=false)" alias=hello-jovinian application=hello-jovinian-aiu-test image_name=PROJECT_ID/MYREPO/hello-python image_tag=0cf9946 registry=us-central1-docker.pkg.dev
time="2024-07-16T18:15:41Z" level=debug msg="Using commit message: build: automatic update of hello-jovinian-aiu-test\n\nupdates image PROJECT_ID/MYREPO/hello-python tag '0cf9946' to 'b0d1559'\n"
[snip]

Here is the hello-thersitean-aiu-test argocd Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: hello-thersitean-aiu-test
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: hello-thersitean-aiu-test=us-central1-docker.pkg.dev/trustle-core-cicd/kallen-test/hello-python
    argocd-image-updater.argoproj.io/hello-thersitean-aiu-test.update-strategy: newest-build
    argocd-image-updater.argoproj.io/hello-thersitean-aiu-test.allow-tags: regexp:^[0-9a-f]{7}$
    argocd-image-updater.argoproj.io/write-back-method: git
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: main
  source:
    repoURL: git@github.com:MYCOMPANY/gitops-poc.git
    targetRevision: poc
    path: aiu/envs/staging/hello-thersitean
  destination:
    server: https://kubernetes.default.svc
    namespace: gcp-control
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m0s
        factor: 2
    syncOptions:
      - CreateNamespace=true

Here is the kustomize and runservice in the gitops repo:

gitops-poc $ cat aiu/hello-thersitean/kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
metadata:
  name: arbitrary
resources:
  - runservice.yaml

gitops-poc $ cat aiu/hello-thersitean/runservice.yaml
apiVersion: run.cnrm.cloud.google.com/v1beta1
kind: RunService
metadata:
  name: hello-thersitean
spec:
  ingress: "INGRESS_TRAFFIC_ALL"
  launchStage: "GA"
  location: "us-central1"
  projectRef:
    external: projects/PROJECT_ID
  template:
    containers:
      - env:
        - name: VOGON
          value: bypass
        - name: ZED
          value: beeble
        #image: us-docker.pkg.dev/cloudrun/container/hello
        image: us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-python:0cf9946
        ports:
        - containerPort: 8080
          name: http1
        resources:
          cpuIdle: true
          limits:
            cpu: 1000m
            memory: 512Mi
    scaling:
      maxInstanceCount: 2
  traffic:
    - percent: 100
      type: "TRAFFIC_TARGET_ALLOCATION_TYPE_LATEST"

To Reproduce In my case, I have a ArgoCD on a GKE cluster with addon Google Config Connector. Apply the Application file, it should launch the Cloud Run service using the initial image tag. Then update the image, pushing a new one to your artifact repo using CI. AIU detects the update image on the repo, but skips it.

Expected behavior I expect AIU does not skip the new image tag, but updates the Cloud Run service using it.

Version argocd-server: v2.11.3+3f344d5 argocd-image-updater v0.13.1

Logs Logs above inline

[chengfang]

See if this helps: https://argocd-image-updater.readthedocs.io/en/latest/configuration/images/#forcing-image-updates

[kallentrustle]

@chengfang that is working! Thank you! But I'm noticing the following behaviors with it.

The RunService app has force-update: "true". When I watch the log, I see the image_tag is empty. See below. is my commentary in the log.

time="2024-07-19T22:18:50Z" level=info msg="Setting new image to us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-python:098771d" alias=hello-cloudrain-aiu-test application=hello-cloudrain-aiu-test image_name=PROJECT_ID/MYREPO/hello-python image_tag=<MISSING> registry=us-central1-docker.pkg.dev
time="2024-07-19T22:18:50Z" level=info msg="Successfully updated image 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-python' to 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-python:098771d', but pending spec update (dry run=false)" alias=hello-cloudrain-aiu-test application=hello-cloudrain-aiu-test image_name=PROJECT_ID/MYREPO/hello-python image_tag=<MISSING> registry=us-central1-docker.pkg.dev
time="2024-07-19T22:18:50Z" level=debug msg="Using commit message: build: automatic update of hello-cloudrain-aiu-test\n\nupdates image PROJECT_ID/MYREPO/hello-python tag '<MISSING>' to '098771d'\n"
time="2024-07-19T22:18:50Z" level=info msg="Committing 1 parameter update(s) for application hello-cloudrain-aiu-test" application=hello-cloudrain-aiu-test

I tested this with a containerized app. Here hello-go uses force-update: "false". image_tag not missing.

time="2024-07-19T22:57:04Z" level=info msg="Setting new image to us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-go:7dc21d1" alias=hello-go application=hello-go image_name=PROJECT_ID/MYREPO/hello-go image_tag=ce102ef registry=us-central1-docker.pkg.dev
time="2024-07-19T22:57:04Z" level=info msg="Successfully updated image 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-go:ce102ef' to 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-go:7dc21d1', but pending spec update (dry run=false)" alias=hello-go application=hello-go image_name=PROJECT_ID/MYREPO/hello-go image_tag=ce102ef registry=us-central1-docker.pkg.dev
time="2024-07-19T22:57:04Z" level=debug msg="Using commit message: build: automatic update of hello-go\n\nupdates image PROJECT_ID/MYREPO/hello-go tag 'ce102ef' to '7dc21d1'\n"
time="2024-07-19T22:57:04Z" level=info msg="Committing 1 parameter update(s) for application hello-go" application=hello-go

Here hello-go uses force-update: "true". There's no difference regarding the presence of the image_tag -- it's present.

time="2024-07-19T23:11:09Z" level=info msg="Setting new image to us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-go:72c945f" alias=hello-go application=hello-go image_name=PROJECT_ID/MYREPO/hello-go image_tag=7dc21d1 registry=us-central1-docker.pkg.dev
time="2024-07-19T23:11:09Z" level=info msg="Successfully updated image 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-go:7dc21d1' to 'us-central1-docker.pkg.dev/PROJECT_ID/MYREPO/hello-go:72c945f', but pending spec update (dry run=false)" alias=hello-go application=hello-go image_name=PROJECT_ID/MYREPO/hello-go image_tag=7dc21d1 registry=us-central1-docker.pkg.dev
time="2024-07-19T23:11:09Z" level=debug msg="Using commit message: build: automatic update of hello-go\n\nupdates image PROJECT_ID/MYREPO/hello-go tag '7dc21d1' to '72c945f'\n"
time="2024-07-19T23:11:09Z" level=info msg="Committing 1 parameter update(s) for application hello-go" application=hello-go