search

argoproj-labs / argocd-image-updater

Automatic container image update for Argo CD
https://argocd-image-updater.readthedocs.io/en/stable/
Apache License 2.0
1.22k stars 253 forks source link

Running image does not get updated #588

Open teekeino opened 1 year ago

teekeino commented 1 year ago

Describe the bug

Running image does not get updated. Image updater logs keeps repeating lines about updating the image, but it does not get updated.

To Reproduce Install versions mentioned, publish a new image. Application configuration below

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: xxx
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd-image-updater.argoproj.io/image-list: api=xxx/api
    argocd-image-updater.argoproj.io/api.update-strategy: latest
spec:
  project: default
  source:
    repoURL: https://gitlab.com/xxx/cloud/infra.git
    targetRevision: HEAD
    path: ./charts/xxx
    helm:
      parameters:
        - name: domain
          value: xx.yyy.ai
  destination:
    server: https://kubernetes.default.svc
    namespace: app
  syncPolicy:
    automated: # automated sync by default retries failed attempts 5 times with following delays between attempts ( 5s, 10s, 20s, 40s, 80s ); retry controlled using `retry` field.
      prune: true # Specifies if resources should be pruned during auto-syncing ( false by default ).
      selfHeal: true # Specifies if partial app sync should be executed when resources are changed only in target Kubernetes cluster and no git change detected ( false by default ).
      allowEmpty: true # Allows deleting all application resources during automatic syncing ( false by default ).

Expected behavior Pod should be recreated with latest image version. Now the Application gets updated somehow but that does not get reflected in the actual deployment.

Additional context When I get details from argocd, there is some info about the image that should get deployed, below.

(venv) 10:53:54: k8s % kubectl -n argocd describe app xxx
Name:         xxx
Namespace:    argocd
Labels:       <none>
Annotations:  argocd-image-updater.argoproj.io/api.pull-secret: argocd/rootsignals-docker-credentials
              argocd-image-updater.argoproj.io/api.update-strategy: latest
              argocd-image-updater.argoproj.io/image-list: api=xxx/api
API Version:  argoproj.io/v1alpha1
Kind:         Application
Metadata:
  Creation Timestamp:  2023-07-07T05:36:21Z
  Finalizers:
    resources-finalizer.argocd.argoproj.io
  Generation:        69
  Resource Version:  3510319
  UID:               bcdc3451-3372-48f2-90c5-ce33fd7f3b1f
Spec:
  Destination:
    Namespace:  app
    Server:     https://kubernetes.default.svc
  Project:      default
  Source:
    Helm:
      Parameters:
        Name:          domain
        Value:         dev.xxx.ai
        Force String:  true
        Name:          image.name
        Value:         xxx/api
        Force String:  true
        Name:          image.tag
        Value:         v0.1.5
    Path:              ./charts/xxx
    Repo URL:          https://gitlab.com/xxx/cloud/infra.git
    Target Revision:   HEAD
  Sync Policy:
    Automated:
      Allow Empty:  true
      Prune:        true
      Self Heal:    true
Status:
  Health:
    Status:  Healthy
  History:
    Deploy Started At:  2023-07-07T05:36:22Z
    Deployed At:        2023-07-07T05:36:22Z
    Id:                 0
    Revision:           fc9c5a31ccc583e725e67bce41a80018fefc8850
    Source:
      Helm:
        Parameters:
          Name:         domain
          Value:        dev.xxx.ai
      Path:             ./charts/xxx
      Repo URL:         https://gitlab.com/xxx/cloud/infra.git
      Target Revision:  HEAD
  Operation State:
    Finished At:  2023-07-07T05:36:22Z
    Message:      successfully synced (all tasks run)
    Operation:
      Initiated By:
        Automated:  true
      Retry:
        Limit:  5
      Sync:
        Prune:     true
        Revision:  fc9c5a31ccc583e725e67bce41a80018fefc8850
        Sync Options:
          CreateNamespace=true
    Phase:       Succeeded
    Started At:  2023-07-07T05:36:22Z
    Sync Result:
      Resources:
        Group:
        Hook Phase:  Running
        Kind:        Service
        Message:     service/flan created
        Name:        flan
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:
        Hook Phase:  Running
        Kind:        Service
        Message:     service/dolly created
        Name:        dolly
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:
        Hook Phase:  Running
        Kind:        Service
        Message:     service/simple-optimizer-service created
        Name:        simple-optimizer-service
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:
        Hook Phase:  Running
        Kind:        Service
        Message:     service/frontend created
        Name:        frontend
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:
        Hook Phase:  Running
        Kind:        Service
        Message:     service/dummy created
        Name:        dummy
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:
        Hook Phase:  Running
        Kind:        Service
        Message:     service/graphql-api created
        Name:        graphql-api
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/dolly created
        Name:        dolly
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/simple-optimizer created
        Name:        simple-optimizer
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/dummy created
        Name:        dummy
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/frontend created
        Name:        frontend
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/flan created
        Name:        flan
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       apps
        Hook Phase:  Running
        Kind:        Deployment
        Message:     deployment.apps/graphql-api created
        Name:        graphql-api
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       networking.k8s.io
        Hook Phase:  Running
        Kind:        Ingress
        Message:     ingress.networking.k8s.io/unified-endpoint created
        Name:        unified-endpoint
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
        Group:       networking.k8s.io
        Hook Phase:  Running
        Kind:        Ingress
        Message:     ingress.networking.k8s.io/app created
        Name:        app
        Namespace:   app
        Status:      Synced
        Sync Phase:  Sync
        Version:     v1
      Revision:      fc9c5a31ccc583e725e67bce41a80018fefc8850
      Source:
        Helm:
          Parameters:
            Name:         domain
            Value:        dev.xxx.ai
        Path:             ./charts/xxx
        Repo URL:         https://gitlab.com/xxx/cloud/infra.git
        Target Revision:  HEAD
  Reconciled At:          2023-07-07T07:54:13Z
  Resources:
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       dolly
    Namespace:  app
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       dummy
    Namespace:  app
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       flan
    Namespace:  app
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       frontend
    Namespace:  app
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       graphql-api
    Namespace:  app
    Status:     Synced
    Version:    v1
    Health:
      Status:   Healthy
    Kind:       Service
    Name:       simple-optimizer-service
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       dolly
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       dummy
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       flan
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       frontend
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       graphql-api
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      apps
    Health:
      Status:   Healthy
    Kind:       Deployment
    Name:       simple-optimizer
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      networking.k8s.io
    Health:
      Status:   Healthy
    Kind:       Ingress
    Name:       app
    Namespace:  app
    Status:     Synced
    Version:    v1
    Group:      networking.k8s.io
    Health:
      Status:   Healthy
    Kind:       Ingress
    Name:       unified-endpoint
    Namespace:  app
    Status:     Synced
    Version:    v1
  Source Type:  Helm
  Summary:
    External UR Ls:
      http://dev.xxx.ai/
      https://ep.dev.xxx.ai/dolly(/|$)(.*)
      https://ep.dev.xxx.ai/flan(/|$)(.*)
      https://ep.dev.xxx.ai/graphql
      https://ep.dev.xxx.ai/simple-optimizer(/|$)(.*)
    Images:
      xxx/api:latest
      xxx/frontend:0.0.13
      xxx/llm_dolly-v2-3b:0.0.3
      xxx/llm_dummy-llm:0.0.4
      xxx/llm_flan-t5-small:0.0.6
      xxx/simple-optimizer:latest
  Sync:
    Compared To:
      Destination:
        Namespace:  app
        Server:     https://kubernetes.default.svc
      Source:
        Helm:
          Parameters:
            Name:          domain
            Value:         dev.xxx.ai
            Force String:  true
            Name:          image.name
            Value:         xxx/api
            Force String:  true
            Name:          image.tag
            Value:         v0.1.5
        Path:              ./charts/xxx
        Repo URL:          https://gitlab.com/xxx/cloud/infra.git
        Target Revision:   HEAD
    Revision:              fc9c5a31ccc583e725e67bce41a80018fefc8850
    Status:                Synced
Events:
  Type    Reason         Age    From                Message
  ----    ------         ----   ----                -------
  Normal  ImagesUpdated  59m    ArgocdImageUpdater  Successfully updated application 'axon1'
  Normal  ImagesUpdated  57m    ArgocdImageUpdater  Successfully updated application 'axon1'
  Normal  ImagesUpdated  55m    ArgocdImageUpdater  Successfully updated application 'axon1'
  Normal  ImagesUpdated  53m    ArgocdImageUpdater  Successfully updated application 'axon1'
  Normal  ImagesUpdated  51m    ArgocdImageUpdater  Successfully updated application 'axon1'
  Normal  ImagesUpdated  49m    ArgocdImageUpdater  Successfully updated application 'axon1'
  Normal  ImagesUpdated  47m    ArgocdImageUpdater  Successfully updated application 'axon1'

Version Kube 1.27 (EKS) ArgoCD 2.7.2 (latest stable also tested) Argocd-image-updater v0.12.2+1aa317c (latest stable also tested)

Logs

time="2023-07-07T06:08:34Z" level=info msg="Committing 1 parameter update(s) for application axon1" application=axon1
time="2023-07-07T06:08:34Z" level=info msg="Successfully updated the live application spec" application=axon1
time="2023-07-07T06:08:34Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=1 errors=0"
time="2023-07-07T06:10:34Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2023-07-07T06:10:34Z" level=warning msg="Invalid credential reference specified: argocd/xxx-docker-credentials" image_alias=api image_name=xxx/api registry_url=
time="2023-07-07T06:10:35Z" level=info msg="Setting new image to xxx/api:v0.1.5" alias=api application=xxx image_name=xxx/api image_tag=latest registry=
time="2023-07-07T06:10:35Z" level=info msg="Successfully updated image 'xxx/api:latest' to 'xxx/api:v0.1.5', but pending spec update (dry run=false)" alias=api application=axon1 image_name=rootsignals/api image_tag=latest registry
teekeino commented 1 year ago

This might duplicate #580. Both update methods (git and direct) fail to update the image even though the git method even makes the commit to the registry.