search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
18.07k stars 5.53k forks source link

ArgoCD server doesn't pick up the new OIDC secret #18576

Open kirgene opened 5 months ago

kirgene commented 5 months ago

Checklist:

Describe the bug

ArgoCD doesn't pick up the new OIDC secret referenced in oidc.config. When the OIDC secret is created dynamically (using external-secrets operator) and doesn't exist at the time argocd-server is first started, ArgoCD can't pick it up.

To Reproduce

  1. Deploy argo-cd

  2. Add this config in argocd-cm configmap:

    oidc.config: |
name: Keycloak
issuer: $argocd-client-secret:oidc.keycloak.issuer
clientID: $argocd-client-secret:oidc.keycloak.clientId
clientSecret: $argocd-client-secret:oidc.keycloak.clientSecret
requestedScopes: ["openid", "profile", "email", "groups"]

  3. In the logs of an argocd-server server, messages should say that the secret argocd-client-secret doesn't exist.

  4. Create a secret argocd-client-secret in the argo-cd namespace:

    kind: Secret
apiVersion: v1
data:
oidc.keycloak.clientId: ZXhhbXBsZQ==
oidc.keycloak.clientSecret: ZXhhbXBsZQ==
oidc.keycloak.issuer: aHR0cHM6Ly9leGFtcGxlLmNvbQ==  # https://example.com
immutable: false
metadata:
labels:
app.kubernetes.io/part-of: argocd
name: argocd-client-secret
namespace: argo-cd
type: Opaque

  5. In the logs of an argocd-server server you'll see that the process gets restarted (the new secret is recognized by argocd-server).

  6. Trying to login with OIDC from argo UI, you'll see the following message:

    Failed to query provider "$argocd-client-secret:oidc.keycloak.issuer": parse "$argocd-client-secret:oidc.keycloak.issuer/.well-known/openid-configuration": first path segment in URL cannot contain colon

    So instead of parsing the referenced secret, argocd-server takes the reference ($argocd-client-secret:oidc.keycloak.issuer) as a value for OIDC issuer.

Expected behavior

ArgoCD should correctly parse the secret references in the configmap for new secrets.

Screenshots

Version

argocd: v2.11.3+3f344d5
  BuildDate: 2024-06-06T09:27:36Z
  GitCommit: 3f344d54a4e0bbbb4313e1c19cfe1e544b162598
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.11.2+25f7504.dirty
  BuildDate: 2024-05-24T03:26:31Z
  GitCommit: 25f7504ecc198e7d7fdc055fdb83ae50eee5edd0
  GitTreeState: dirty
  GoVersion: go1.21.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.4+g81c902a
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0

Logs

Paste any relevant application logs here.
agaudreault commented 5 months ago

@kirgene do you have time to investigate this deeper?

What do you mean at step 4 In the logs of an argocd-server server you'll see that the process gets restarted (the new secret is recognized by argocd-server).. Can you share these logs?

From the code, it seems like the server oidc client will not reflect updates to the settings/secrets after the creation.

It might also be related to https://github.com/argoproj/argo-cd/issues/18269

kirgene commented 5 months ago

@agaudreault here is a complete workflow:

  1. Run argo without OIDC secret: 
    time="2024-06-22T14:07:39Z" level=info msg="ArgoCD API Server is starting" built="2024-05-24T03:26:31Z" commit=25f7504ecc198e7d7fdc055fdb83ae50eee5edd0 namespace=argo-cd port=8080 version=v2.11.2+25f7504.dirty
time="2024-06-22T14:07:39Z" level=info msg="Starting configmap/secret informers"
time="2024-06-22T14:07:39Z" level=info msg="Configmap/secret informer synced"
time="2024-06-22T14:07:39Z" level=info msg="admin disabled"
time="2024-06-22T14:07:39Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientId', but key does not exist in secret"
time="2024-06-22T14:07:39Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientSecret', but key does not exist in secret"
time="2024-06-22T14:07:39Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.issuer', but key does not exist in secret"
time="2024-06-22T14:07:39Z" level=info msg="invalidated cache for resource in namespace: argo-cd with the name: argocd-notifications-cm"
time="2024-06-22T14:07:39Z" level=info msg="invalidated cache for resource in namespace: argo-cd with the name: argocd-notifications-secret"
time="2024-06-22T14:07:39Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientId', but key does not exist in secret"
time="2024-06-22T14:07:39Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientSecret', but key does not exist in secret"
time="2024-06-22T14:07:39Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.issuer', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=info msg="Creating client app ($argocd-client-secret:oidc.keycloak.clientId)"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientId', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientSecret', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.issuer', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientId', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientSecret', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.issuer', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=info msg="argocd v2.11.2+25f7504.dirty serving on port 8080 (url: https://internal.domain.com/argocd, tls: false, namespace: argo-cd, sso: true)"
time="2024-06-22T14:07:40Z" level=info msg="Enabled application namespace patterns: argo-cd, argo-cd, argo-cd-dev, argo-cd-stage"
time="2024-06-22T14:07:40Z" level=info msg="0xc00109de60 subscribed to settings updates"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientSecret', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.issuer', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=warning msg="config referenced '$argocd-client-secret:oidc.keycloak.clientId', but key does not exist in secret"
time="2024-06-22T14:07:40Z" level=info msg="Starting rbac config informer"
time="2024-06-22T14:07:40Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
  2. Try to login with OIDC (notice the issuer value): In the server logs: 
    time="2024-06-22T14:23:27Z" level=info msg="Initializing OIDC provider (issuer: $argocd-client-secret:oidc.keycloak.issuer)"

    On the web page:

    Failed to query provider "$argocd-client-secret:oidc.keycloak.issuer": parse "$argocd-client-secret:oidc.keycloak.issuer/.well-known/openid-configuration": first path segment in URL cannot contain colon
  3. Create OIDC secret, this change is recognized by argocd (it gets reloaded): 
    time="2024-06-22T14:33:01Z" level=info msg="Notifying 1 settings subscribers: [0xc00109de60]"
time="2024-06-22T14:33:01Z" level=info msg="oidc config modified. restarting"
time="2024-06-22T14:33:01Z" level=info msg="shutting down settings watch"
time="2024-06-22T14:33:01Z" level=info msg="Shut down requested"
time="2024-06-22T14:33:01Z" level=info msg="0xc00109de60 unsubscribed from settings updates"
time="2024-06-22T14:33:01Z" level=info msg="rbac configmap informer cancelled"
time="2024-06-22T14:33:01Z" level=info msg="Creating client app (6559d9c0-a6db-443b-bb0f-fbb3a3e78c9d)"
time="2024-06-22T14:33:01Z" level=info msg="argocd v2.11.2+25f7504.dirty serving on port 8080 (url: https://internal.domain.com/argocd, tls: false, namespace: argo-cd, sso: true)"
time="2024-06-22T14:33:01Z" level=info msg="Enabled application namespace patterns: argo-cd, argo-cd, argo-cd-dev, argo-cd-stage"
time="2024-06-22T14:33:01Z" level=info msg="0xc000c575c0 subscribed to settings updates"
time="2024-06-22T14:33:01Z" level=info msg="Notifying 1 settings subscribers: [0xc000c575c0]"
time="2024-06-22T14:33:01Z" level=info msg="Starting rbac config informer"
time="2024-06-22T14:33:01Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
time="2024-06-22T14:33:02Z" level=info msg="Notifying 1 settings subscribers: [0xc000c575c0]"
time="2024-06-22T14:33:04Z" level=info msg="Notifying 1 settings subscribers: [0xc000c575c0]"
time="2024-06-22T14:33:05Z" level=info msg="Notifying 1 settings subscribers: [0xc000c575c0]"
  4. Try to login with OIDC (notice the issuer value): In the server logs: 
    time="2024-06-22T14:36:37Z" level=info msg="Initializing OIDC provider (issuer: $argocd-client-secret:oidc.keycloak.issuer)"

    On the web page:

    Failed to query provider "$argocd-client-secret:oidc.keycloak.issuer": parse "$argocd-client-secret:oidc.keycloak.issuer/.well-known/openid-configuration": first path segment in URL cannot contain colon
  5. Restart argocd server: 
    time="2024-06-22T14:38:50Z" level=info msg="ArgoCD API Server is starting" built="2024-05-24T03:26:31Z" commit=25f7504ecc198e7d7fdc055fdb83ae50eee5edd0 namespace=argo-cd port=8080 version=v2.11.2+25f7504.dirty
time="2024-06-22T14:38:50Z" level=info msg="Starting configmap/secret informers"
time="2024-06-22T14:38:50Z" level=info msg="Configmap/secret informer synced"
time="2024-06-22T14:38:50Z" level=info msg="admin disabled"
time="2024-06-22T14:38:50Z" level=info msg="invalidated cache for resource in namespace: argo-cd with the name: argocd-notifications-secret"
time="2024-06-22T14:38:50Z" level=info msg="invalidated cache for resource in namespace: argo-cd with the name: argocd-notifications-cm"
time="2024-06-22T14:38:50Z" level=info msg="Creating client app (6559d9c0-a6db-443b-bb0f-fbb3a3e78c9d)"
time="2024-06-22T14:38:50Z" level=info msg="argocd v2.11.2+25f7504.dirty serving on port 8080 (url: https://internal.domain.com/argocd, tls: false, namespace: argo-cd, sso: true)"
time="2024-06-22T14:38:50Z" level=info msg="Enabled application namespace patterns: argo-cd, argo-cd, argo-cd-dev, argo-cd-stage"
time="2024-06-22T14:38:50Z" level=info msg="0xc001121440 subscribed to settings updates"
time="2024-06-22T14:38:50Z" level=info msg="Starting rbac config informer"
time="2024-06-22T14:38:50Z" level=info msg="RBAC ConfigMap 'argocd-rbac-cm' added"
  6. Try to login with OIDC (notice the issuer value): 
    time="2024-06-22T14:39:48Z" level=info msg="Initializing OIDC provider (issuer: https://auth.internal.domain.com/realms/test)"
time="2024-06-22T14:39:48Z" level=info msg="OIDC supported scopes: [openid roles acr profile phone microprofile-jwt offline_access web-origins groups email address]"
time="2024-06-22T14:39:48Z" level=info msg="Performing authorization_code flow login: https://auth.internal.domain.com/realms/test/protocol/openid-connect/auth...."

After the restart you can see the issuer value is correct and OIDC login works.

agaudreault commented 5 months ago

@kirgene Thank you for the logs. It would seem that the OIDC provider will create a new internal instance of the provider, but it will reuse the initial configuration.

The following function is the one parasing the secret as you can see with "Creating client app ($argocd-client-secret:oidc.keycloak.clientId)", it does not have the correct value at the time of creation and will not be recreate when settings changes. https://github.com/argoproj/argo-cd/blob/0351e9d50995884daf8b9932eb497bc9fa93d523/util/oidc/oidc.go#L95

svghadi commented 5 months ago

I have encountered a similar issue in #14038 where the SSO configurations are not updated when switching from Dex to Keycloak. During my investigation at that time, I found that the hot-reload functionality of the ArgoCD server appears to be the culprit.

https://github.com/argoproj/argo-cd/issues/14038#issuecomment-1657503131

While debugging, I found that during hot-reload, after the new server object is created, we do not shutdown the older server object, causing the incoming HTTP traffic to be distributed amongst these server objects in an indeterministic way, which leads to such behaviours. Shouldn't the older server be shutdown/cleaned once new server is up?

todaywasawesome commented 5 months ago

Confirmed as a bug by @alexmt - difficult to debug as it isn't reproducible on local clusters. We run into it often with Github authentication where it takes a few tries and they get 404 because an app id goes in and isn't updated properly. Eventually, it will resolve itself without any intervention, there is a go routine that watches for updates and triggers to reload settings.

Alex will share link to the code area.

dfry commented 1 month ago

Any updates on this?

OpenGuidou commented 1 month ago

Hi, I managed to reproduce that on local cluster. In my case it doesn't resolve itself. I assigned an int ID to the server and a few logs to make it clearer:

Steps:

  1. Assign a random ID to the api-server in the code when it starts

  2. Create an Argocd-secret with fake data:

    apiVersion: v1
kind: Secret
metadata:
name: argocd-secret
namespace: argocd
labels:
app.kubernetes.io/name: argocd-secret
app.kubernetes.io/part-of: argocd
type: Opaque
data:
oidc.myoidc.issuer: aHR0cDovL29pZGMuMTI3LjAuMC4xLm5pcC5pbwo=
oidc.myoidc.clientId: "bXlOZXdDbGllbnRJZAo="
# mySecret
oidc.myoidc.clientSecret: "bXlTZWNyZXQK"
# myNewSecret
#oidc.myoidc.clientSecret: "bXlOZXdTZWNyZXQK"

  3. Create an argocd-cm with oidc config referencing the secret

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  oidc.config: |
      name: MyOIDC
      issuer: $oidc.myoidc.issuer
      clientID: $oidc.myoidc.clientId
      clientSecret: $oidc.myoidc.clientSecret
  1. Update the secret (switch from mySecret to myNewSecret)
  2. See an extract of the logs below:
api-server | INFO[0000] Starting new server with id 3
...
api-server | INFO[0000] argocd v99.99.99+unknown serving on port 8080 (url: , tls: false, namespace: argocd, sso: true)
...
// Perform login action
api-server | INFO[0077] Serving login from server 3
api-server | INFO[0077] Initializing OIDC provider (issuer: http://oidc.127.0.0.1.nip.io)
...
//Update configmap
api-server | INFO[0093] oidc config modified. restarting
api-server | INFO[0093] shutting down settings watch
api-server | INFO[0093] 0xc00148f620 unsubscribed from settings updates 
api-server | INFO[0093] rbac configmap informer cancelled
api-server | INFO[0093] Starting new server with id 88
...
api-server | INFO[0093] Creating client app (myNewClientId)
...
// Perform login action
api-server | INFO[0121] Serving login from server 3

As you can see, server 3 which was supposed to be shutdown is still serving the login action. It would explain why the OIDC config is still the outdated one.