dtaniwaki commented 5 years ago

Describe the bug

Although the comment in secret.yaml indicates server.secretkey can be auto-generated if it's missing. https://github.com/argoproj/argo-cd/blob/d3c850b8e7e67d1aa4c2deb6b77a4edbf4b7f261/docs/operator-manual/argocd-secret.yaml#L19

However, the code doesn't allow empty server.secretkey.

https://github.com/argoproj/argo-cd/blob/d3c850b8e7e67d1aa4c2deb6b77a4edbf4b7f261/util/settings/settings.go#L499

Which is the expected behavior?

To Reproduce

Just after installing ArgoCD and set admin.password (although this is not mentioned in the docs).

Expected behavior

Asking the expected behavior.

Screenshots

N/A

Version

$ argocd version
argocd: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:36Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.0.2+e0bd546.dirty
  BuildDate: 2019-06-14T17:15:03Z
  GitCommit: e0bd546a07818ec06a27c2b3033454e3eb1c4152
  GitTreeState: dirty
  GoVersion: go1.11.4
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: 0.13.1

Logs

time="2019-07-14T11:45:39Z" level=info msg="Starting configmap/secret informers"
time="2019-07-14T11:45:39Z" level=info msg="Configmap/secret informer synced"
time="2019-07-14T11:45:39Z" level=fatal msg="server.secretkey is missing"

Have you thought about contributing a fix yourself?

I'm not sure which is the expected behavior.

jessesuen commented 5 years ago

The API server is supposed to generate this. During new installations, we will sometimes see application-controller crash because it started before the api-server had a chance to generate it.

Which service are your logs coming from?

dtaniwaki commented 5 years ago

The log came from the application controller.

dtaniwaki commented 5 years ago

Uh, the api server was not running because of PSP. I'll try it and update you again.

dtaniwaki commented 5 years ago

I confirmed it is created automatically if the api server is running.

emidevi commented 2 years ago

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

alkdese commented 2 years ago

Issue: The server.secretkey was missing / did not generate on its own.

What we tried: We took the value of server.secretkey from our production environment and assigned it on our E1 environment where it was missing.

Output: We were getting error as unauthorized and we had no idea why.

Ultimate Solution: Uninstalled and reinstalled argo from scratch and then it generated the server.secretkey on its own.

So the server.secretkey has to have a unique value and should be generated on its own. We cannot assign a value of our own.

In my situation I started having CrashLoopBackoff on argocd-dex-server in the logs I have "server.secretkey is missing" after deleting argocd namespace and creating it from scratch. I just now deleted already 3 times and recreated and still same problem. Any advice?

hyerim-gn commented 2 years ago

I solved the problem with node scale out due to the problem caused by the API server not being executed.

emirot commented 2 years ago

Seeing this issue after doing:

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.4.7/manifests/install.yaml

Then CrashLoopBackOff on argocd-dex-server

kubectl logs argocd-dex-server-56d495844c-l2lxc -n argocd
time="2022-07-20T22:45:06Z" level=info msg="Starting configmap/secret informers"
time="2022-07-20T22:45:06Z" level=info msg="Configmap/secret informer synced"
time="2022-07-20T22:45:06Z" level=fatal msg="server.secretkey is missing"

crenshaw-dev commented 2 years ago

@emirot can you confirm that the API server is successfully coming up?

emirot commented 2 years ago

@crenshaw-dev sorry about the noise, I figured it out, it's not related, it is due to Open Policy Agent

brianpooe commented 2 years ago

@emirot i have the exact same issue how did you resolve yours?

emirot commented 2 years ago

@brianpooe In my case I had to do add to add --set global.networkPolicy.create=true in my helm install

winston0410 commented 2 years ago

I have the same issue when I install argocd with Core Install. @dtaniwaki Can we reopen this issue?

yevhen-harmonizehr commented 2 years ago

i do have same issue after my redis pods restarted. My argo redis do not have PVC, so this might be related.

TeamDman commented 2 years ago

10831 might be related

I had this

$ kubectl get all -n argocd
NAME                                                 READY   STATUS             RESTARTS      AGE
pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx   1/1     Running            0             17m
pod/argocd-argo-cd-repo-server-9565b9f86-5956t       1/1     Running            0             17m
pod/argocd-argo-cd-server-685dfc4dd7-kgl8r           0/1     CrashLoopBackOff   7 (33s ago)   11m
pod/argocd-redis-master-0                            1/1     Running            0             12m

NAME                                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/argocd-argo-cd-app-controller   ClusterIP   10.0.189.90   <none>        8082/TCP         17m
service/argocd-argo-cd-repo-server      ClusterIP   10.0.80.110   <none>        8081/TCP         17m
service/argocd-argo-cd-server           ClusterIP   10.0.42.249   <none>        80/TCP,443/TCP   17m
service/argocd-redis-headless           ClusterIP   None          <none>        6379/TCP         17m
service/argocd-redis-master             ClusterIP   10.0.51.188   <none>        6379/TCP         17m

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-argo-cd-app-controller   1/1     1            1           17m
deployment.apps/argocd-argo-cd-repo-server      1/1     1            1           17m
deployment.apps/argocd-argo-cd-server           0/1     1            0           17m

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-argo-cd-app-controller-6fbb4ccf8d   1         1         1       17m
replicaset.apps/argocd-argo-cd-repo-server-9565b9f86       1         1         1       17m
replicaset.apps/argocd-argo-cd-server-685dfc4dd7           1         1         0       17m

NAME                                   READY   AGE
statefulset.apps/argocd-redis-master   1/1     17m

$ kubectl logs -n argocd pod/argocd-argo-cd-server-685dfc4dd7-kgl8r
Defaulted container "argocd-server" out of: argocd-server, wait-for-redis (init)
 19:37:13.78 
 19:37:13.79 Welcome to the Bitnami argo-cd container
 19:37:13.79 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:37:13.79 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:37:13.79 
 19:37:13.80 INFO  ==> Configuring libnss_wrapper

time="2022-11-01T19:37:13Z" level=info msg="ArgoCD API Server is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd port=8080 version=v99.99.99+unknown
time="2022-11-01T19:37:13Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="Initialized server signature"
time="2022-11-01T19:37:14Z" level=info msg="admin disabled"
time="2022-11-01T19:37:14Z" level=info msg="Starting configmap/secret informers"
time="2022-11-01T19:37:14Z" level=info msg="configmap informer cancelled"
time="2022-11-01T19:37:14Z" level=info msg="Configmap/secret informer synced"
time="2022-11-01T19:37:14Z" level=info msg="secrets informer cancelled"
panic: server.secretkey is missing

goroutine 1 [running]:
github.com/argoproj/argo-cd/v2/util/session.NewSessionManager(0xc00004d980, {0x44c4aa0, 0xc000fcc580}, {0x34aec23, 0x16}, 0xc000aee5e0?, {0x44ddf10, 0xc0005a1f80})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/util/session/sessionmanager.go:124 +0x3fc
github.com/argoproj/argo-cd/v2/server.NewServer({_, _}, {0x0, 0x0, 0x1, {0x7ffc91ef54aa, 0x18}, 0x1f90, 0x1f93, {0xc000aee5e0, ...}, ...})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/server/server.go:267 +0x5af
github.com/argoproj/argo-cd/v2/cmd/argocd-server/commands.NewCommand.func1(0xc000afea00?, {0x347c98d?, 0xb?, 0xb?})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/argocd-server/commands/argocd_server.go:192 +0xfa8
github.com/spf13/cobra.(*Command).execute(0xc000afea00, {0xc0000ba010, 0xb, 0xb})
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:876 +0x67b
github.com/spf13/cobra.(*Command).ExecuteC(0xc000afea00)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:990 +0x3b4
github.com/spf13/cobra.(*Command).Execute(...)
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/pkg/mod/github.com/spf13/cobra@v1.5.0/command.go:918
main.main()
        /bitnami/blacksmith-sandox/argo-cd-2.5.0/src/github.com/argoproj/argo-cd/cmd/main.go:57 +0x2c5

$ kubectl logs -n argocd pod/argocd-argo-cd-app-controller-6fbb4ccf8d-7dpnx
Defaulted container "controller" out of: controller, wait-for-redis (init)
 19:21:17.44 
Welcome to the Bitnami argo-cd container
 19:21:17.44 Subscribe to project updates by watching https://github.com/bitnami/containers
 19:21:17.45 Submit issues and feature requests at https://github.com/bitnami/containers/issues
 19:21:17.45 
INFO  ==> Configuring libnss_wrapper
level=info msg="ArgoCD Application Controller is starting" built="1970-01-01T00:00:00Z" commit= namespace=argocd version=v99.99.99+unknown
level=info msg="Processing all cluster shards"
level=info msg="appResyncPeriod=3m0s, appHardResyncPeriod=0s"
level=info msg="Starting configmap/secret informers"
level=info msg="Configmap/secret informer synced"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=info msg="0xc000ff02a0 subscribed to settings updates"
level=info msg="Starting secretInformer forcluster"
level=warning msg="Failed to save clusters info: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: secret \"argocd-secret\" not found"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"
level=warning msg="Failed to save clusters info: server.secretkey is missing"
level=info msg="Notifying 1 settings subscribers: [0xc000ff02a0]"
level=warning msg="Unable to parse updated settings: server.secretkey is missing"

Workaround

Manually specifying server.secretKey as an empty string in argocd-secret seems to have fixed the problem without having to manually restart anything after helm applying... idk if everything is actually working, but the pod stopped crashing and logs look healthy now

gaeljw commented 1 year ago

For anyone coming here for similar issue where server.secretKey is not defined, here's what I did to make it work back:

delete the secret
re-create the secret but empty
restart argocd-server

ali-de7 commented 1 year ago

I fixed it by simply restarting the server. Here's what I did to fix it:

kubectl rollout restart deploy/argocd-server

crenshaw-dev commented 1 year ago

Reopening because I think this is almost definitely a persistent problem with core install.

pgpx commented 1 year ago

Could we split argocd-secret into 2 or more secrets? Leave one for the autogenerated values, another for webhook secrets, and maybe another for the admin password (which is optionally autogenerated)? I'm encountering the same issue trying to introduce a webhook.gitlab.secret via Helm. And/or use server-side-apply for the autogenerated values?

GuillaumeTC1 commented 11 months ago

Hello, I face a related issue on ArgoCD 2.9 managed with Helm Charts. Argo goes out of sync because of required secrets config. If I sync I loose access right away and need to re add manually secrets in Azure.

Capture d’écran 2023-12-11 150738

Am I the only one to encounter this issue ? Do you think it would be possible to fix this without having to ignore argocd-secret sync in argo-cm ?

jdubs commented 10 months ago

I'm also having this same issue with self managed argo. This virtually makes the api/ui not usable, it makes me think that it should fail the health check so that it can be restarted automatically with the new secret.

fstr commented 9 months ago

I ran into the same issue as https://github.com/argoproj/argo-cd/issues/1936#issuecomment-1850161571 It's also simple to reproduce together with the ArgoCD Helm chart and ArgoCD managing itself.

Add a github.webhook.secret:

argo-cd:
  configs:
    secret:
       githubSecret: foobar

Sync it using ArgoCD. Successfully setup. Now your secret will look like this or similar:

apiVersion: v1
data:
  accounts.argoreadonly.tokens: +++++
  admin.password: +++++
  admin.passwordMtime: +++++
  server.secretkey: +++++
  webhook.github.secret: +++++

Now remove the github.webhook.secret by removing the configs: block from your value file again. The argocd-secret will now have an empty data: block when you run helm template because you removed the only value managed by the chart. If you now go into ArgoCD and sync, it will remove all the properties within data property of the argocd-secret. Including all those auto-generated values from the server. After that, ArgoCD stops working. You need to follow the steps from https://github.com/argoproj/argo-cd/issues/1936#issuecomment-1318892424 to resurrect it.

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in https://github.com/argoproj/argo-cd/pull/16262 obsolete.

MrFreezeex commented 9 months ago

I agree with @pgpx that the webhook secrets should be moved into their own Kubernetes Secret. This will not only solve the issue that many people in here are facing, but also make workarounds like the one currently built and proposed by @MrFreezeex in #16262 obsolete.

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

fstr commented 9 months ago

Note that my PR is not fully related to your current issue my use case was more about using https://external-secrets.io/latest/ to supply webhook secrets because that's what we use in our env but that's pretty nice if it solves other similar-ish issues as well! I also believe that having what you are proposing (a separate secret) and my PR are fully compatible and should allow even better flexibility combined :D.

How I got into this topic is also by the need to provide the secret via external-secrets. If the webhook secret would be stored in a separate secret, then we could simply provision that via external-secrets and there's no more need to reference it via the special $<secret-name>.<property> syntax. Your PR will bridge the gap until then though.

RobinsonZ commented 5 months ago

This is also causing an issue for us when argocd-secret is itself managed by argocd, and (as mentioned above) it's compounded by the fact that the argocd-server healthcheck will still succeed even when this error is occurring and making the interface unusable.

gerardnico commented 1 month ago

I got this problem after using external-secrets to provide a github webhook token

In case, you are in the same situation, the workaround is to:

use the creationPolicy: 'Merge'. Example:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: argocd-secret
spec:
secretStoreRef:
name: vault-external-secret-store
kind: ClusterSecretStore
target:
name: argocd-secret # Secret name in Kubernetes
creationPolicy: 'Merge'
template:
  data:
    webhook.github.secret: "{{ .github_webhook_token }}"
  metadata:
    annotations:
      description: "The WebHook Secret that permits to authenticate the GitHub Webhook"
# Mapping to local secret from remote secret
data:
# https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/#2-configure-argo-cd-with-the-webhook-secret-optional
- secretKey: github_webhook_token # Prop Name in the secret
  remoteRef:
    key: argocd # Name of the remote secret
    property: github-webhook-token # Prop Name in the remote secret

Rerun the install to get a clean argocd-secret
Restart the argo-server deployment to get the missing secrets

argoproj / argo-cd

Secret key can not be omitted #1936

10831 might be related

Workaround