ArgoCD upgrade from Helm Chart 6.7.3 to 7.3.11 fails due to argo-redis secret job

bhavinkotak commented 3 months ago

Describe the bug

We have deployed ArgoCD in AKS Cluster 1.27. Redis-HA mode is enabled from argo-cd helm chart. Argo is getting self managed. So installation/upgrade is done by updating ArgoCD app-set.yaml and not via helm upgrade command.

We tried upgrading ArgoCD using helm chart from v6.7.3 to v7.3.11. It tries to upgrade itself; however, there is a job (Name: argocd-redis-secret-init; Kind: Job) which just hangs.

argocd-redis-secret-init-xxxxx POD logs -

Checking for initial Redis password in secret argocd/argocd-redis at key auth.
Argo CD Redis secret state confirmed: secret name argocd-redis.
Password secret is configured properly.

We are able to see new secret getting created. However, no other error message or information gets posted. We did restart all the deployment and sts resources for argocd; however no changes are observed.

Related helm chart

argo-cd

Helm chart version

7.3.11

To Reproduce

Install ArgoCD helm chart v6.7.3 - everything is up and running
Upgrade ArgoCD to helm chart v7.3.11 - it tries to upgrade itself; however there is a job (Name: argocd-redis-secret-init; Kind: Job) which just hangs.

argocd-redis-secret-init-xxxxx POD logs -

Checking for initial Redis password in secret argocd/argocd-redis at key auth. Argo CD Redis secret state confirmed: secret name argocd-redis. Password secret is configured properly.

We are able to see new secret getting created. However, no other error message or information gets posted. We did restart all the deployment and sts resources for argocd; however no changes are observed.

Expected behavior

Successful upgrade of ArgoCD using latest helm chart 7.3.11

Screenshots

No response

Additional context

No response

AssenDimitrov commented 3 months ago

Same, you got any workaround? I am using argocd in GKE, without network policies...

bhavinkotak commented 3 months ago

nah.. I am still stuck on this issue...

jkleinlercher commented 3 months ago

Does the job hang in PreSync hook in ArgoCD UI? I have the same problem posted in https://github.com/argoproj/argo-cd/issues/6880#issuecomment-2263802072

jkleinlercher commented 3 months ago

I think a ttl for the redis job could help

yu-croco commented 3 months ago

Maybe fixed in https://github.com/argoproj/argo-helm/pull/2861 ? Please feel free to reopen if situation is not fixed yet.

speedythesnail commented 2 months ago

This issue should be re-opened. I am unable to do a fresh install of ArgoCD using the helm chart. I am able to do so using the operator in OpenShift, but in my Kind cluster it fails using the chart.

keithdwilliams commented 2 months ago

Facing the same issue, this needs to be re-opened.

lboclboc commented 2 months ago

Facing the same issue with the helm chart version 7.5.2. The job for creating the argocd-redis secret is not run at so several containers is complaining about the missing secret.

DoumLaberge commented 1 month ago

I have a similar issue when I try ton install the Argocd Helm Chart version 7.5.1. I've try the 6.11.1 version and the problem is al;ready there. I've the redis HA mode. I have this error:

Error: UPGRADE FAILED: pre-upgrade hooks failed: warning: Hook pre-upgrade argo-cd/templates/redis-secret-init/job.yaml failed: Job in version "v1" cannot be handled as a Job: json: cannot unmarshal string into Go struct field PodSpec.spec.template.spec.imagePullSecrets of type []v1.LocalObjectReference

when I run it in debug mode, I can see there somthing not right in the argo-cd/templates/redis-secret-init/job.yaml There are a white line between the key iomagePullSecrets and the value


# Source: argo-cd/templates/redis-secret-init/job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: argocd-redis-secret-init
  namespace: "argocd"
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-delete-policy": before-hook-creation
  labels:
    helm.sh/chart: argo-cd-7.5.2
    app.kubernetes.io/name: argocd-redis-secret-init
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/component: redis-secret-init
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: "v2.12.3"
spec:
  ttlSecondsAfterFinished: 60
  template:
    metadata:
      labels:
        helm.sh/chart: argo-cd-7.5.2
        app.kubernetes.io/name: argocd-redis-secret-init
        app.kubernetes.io/instance: argocd
        app.kubernetes.io/component: redis-secret-init
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/part-of: argocd
        app.kubernetes.io/version: "v2.12.3"
    spec:
      imagePullSecrets:

        nexussecret
      containers:
      - command:
          - argocd
          - admin
          - redis-initial-password
        image: quay.io/argoproj/argocd:v2.12.3
        imagePullPolicy: IfNotPresent
        name: secret-init
        resources:
          {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
      restartPolicy: OnFailure
      serviceAccountName: argocd-redis-secret-init

DoumLaberge commented 1 month ago

The template redis-secret-init/job.yaml was introduce in 6.10.0 and doesn't seem to work since

DoumLaberge commented 1 month ago

I just find the error

{{- with .Values.global.imagePullSecrets }} imagePullSecrets: {{ toYaml . | nindent 8 }} {{- end }}

There a missing '-' in front of the toYaml in the redis-secret-init/job.yaml

eltalkarim commented 1 month ago

This is still happening

~ helm upgrade argocd argo/argo-cd --version 7.6.5 --wait -n argocd --debug 
upgrade.go:142: [debug] preparing upgrade for argocd
upgrade.go:524: [debug] copying values from argocd (v14) to new release.
upgrade.go:150: [debug] performing update for argocd
upgrade.go:322: [debug] creating upgraded release for argocd
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" ServiceAccount
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" Role
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" RoleBinding
client.go:128: [debug] creating 1 resource(s)
client.go:310: [debug] Starting delete for "argocd-redis-secret-init" Job
client.go:128: [debug] creating 1 resource(s)
client.go:540: [debug] Watching for changes to Job argocd-redis-secret-init with timeout of 5m0s
client.go:568: [debug] Add/Modify event for argocd-redis-secret-init: ADDED
client.go:607: [debug] argocd-redis-secret-init: Jobs active: 1, jobs failed: 0, jobs succeeded: 0
upgrade.go:434: [debug] warning: Upgrade "argocd" failed: pre-upgrade hooks failed: timed out waiting for the condition
Error: UPGRADE FAILED: pre-upgrade hooks failed: timed out waiting for the condition
helm.go:84: [debug] pre-upgrade hooks failed: timed out waiting for the condition

Logs from the init job:

│ argocd-redis-secret-init-jc9jp Checking for initial Redis password in secret argocd/argocd-redis at key auth.                                                                                                                                                                │
│ argocd-redis-secret-init-jc9jp Argo CD Redis secret state confirmed: secret name argocd-redis.                                                                                                                                                                               │
│ argocd-redis-secret-init-jc9jp Password secret is configured properly.

using the default values here.

argoproj / argo-helm