agilgur5
commented
5 months ago Thanks for the detailed report!
In a Slack convo Anton suggested that #12573 may be suspect.
Specifically, the only SSO change in v3.5.5 was #12318, which is unrelated in this case (different provider, different claim). So my suspicion would then be that a deps change somehow impacted this, and #12573 upgraded expr which is used for rbac-rule evaluation here and is the part with the error:
[...] failed to evaluate rule: unknown name groups (1:43)\n | 'ac1ef805-ac6a-4ce9-854a-1cb406aa7121' in groups\n | ..........................................^"
@Freddybob4244 could you provide a few more details that I had asked for in my last message:
- The SA you were trying to match with the
rbac-ruleannotation you were using here - To clarify, is
ac1ef805-ac6a-4ce9-854a-1cb406aa7121actually one of your groups?
Admin users can navigate normally.
Also can you provide the admin SA with its rbac-rule as well? I'm curious how it differs since it applies correctly.
Happy to provide any more information on request or connect to experiment/work through the issue
From the same message, could you try some of these options and check what the logs say:
- claims other than
groups - expressions other than
in
I'm not sure if we have a robust test environment for SSO issues; we do have an optional Dex but I'm not sure if it's configured and used for SSO tests (I don't think it is IIRC). That would be great to have for reproducibility and to add test cases for the pieces of SSO that are not provider specific.
antonmedv
tooptoop4
ianmuge
Freddybob4244
thunder-spb
Pre-requisites
:latestimage tag (i.e.quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on:latest. If not, I have explained why, in detail, in my description below.What happened/what you expected to happen?
Summary
All non-admin users assume a read-only role. After v3.5.5 is applied, non-admin users can no longer use argo-workflows and are prompted with a generic error in the UI:
Admin users can navigate normally.
Argo-server pod logs provide a bit more useful insight:
There aren't any additional logs that are related to this error that I can find.
Testing Performed
To confirm 3.5.5 introduced the issue I tested a few different versions with a read-only test user in a sandbox installation.
Configurations
```yaml apiVersion: apps/v1 kind: Deployment metadata: name: argo-server namespace: argo spec: selector: matchLabels: app: argo-server template: metadata: labels: app: argo-server spec: containers: - args: - server - "--auth-mode" - sso - "--auth-mode" - client - "--namespaced" - "--secure=false" env: [] image: quay.io/argoproj/argocli:3.5.6 name: argo-server ports: - containerPort: 2746 name: web readinessProbe: httpGet: path: / port: 2746 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 20 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true volumeMounts: - mountPath: /tmp name: tmp nodeSelector: kubernetes.io/os: linux securityContext: runAsNonRoot: true serviceAccountName: argo-server volumes: - emptyDir: {} name: tmp ```Argo-Server
```yaml apiVersion: v1 kind: ConfigMap metadata: name: workflow-controller-configmap namespace: argo data: artifactRepository: | archiveLogs: true s3: bucket: logs-devops endpoint: minio:9000 insecure: true accessKeySecret: name: my-minio-cred key: accesskey secretKeySecret: name: my-minio-cred key: secretkey links: | - name: Example Workflow Link scope: workflow url: http://logging-facility?namespace=${metadata.namespace}&workflowName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt} - name: Example Pod Link scope: pod url: http://logging-facility?namespace=${metadata.namespace}&podName=${metadata.name}&startedAt=${status.startedAt}&finishedAt=${status.finishedAt} metricsConfig: | disableLegacy: true enabled: true path: /metrics port: 9090 persistence: | connectionPool: maxIdleConns: 100 maxOpenConns: 0 connMaxLifetime: 0s nodeStatusOffLoad: true archive: true archiveTTL: 30d postgresql: ssl: true sslmode: require host: ****.postgres.database.azure.com port: 5432 database: argodevops tableName: argo_workflows userNameSecret: name: argo-postgres-creds key: username passwordSecret: name: argo-postgres-creds key: password sso: > issuer: https://login.microsoftonline.com/****/v2.0 clientId: name: client-id-secret key: client-id-key clientSecret: name: client-secret-secret key: client-secret-key redirectUrl: https://argo-devops.dev.int.****.cloud/oauth2/callback scopes: - https://graph.microsoft.com/Group.Read.All rbac: enabled: true workflowDefaults: | spec: activeDeadlineSeconds: 14400 ttlStrategy: secondsAfterCompletion: 604800 podGC: strategy: OnWorkflowCompletion nodeSelector: agentpool: "argoeph" tolerations: - key: workload-type operator: Equal value: argo effect: NoSchedule ```workflow-controller-configmap
```yaml apiVersion: v1 kind: ServiceAccount metadata: name: argo-server namespace: argo ```Server Service Account
```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: argo-server-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: argo-server-cluster-role subjects: - kind: ServiceAccount name: argo-server namespace: argo ```Server ClusterRoleBinding
```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: argo-server-cluster-role rules: - apiGroups: - "" resources: - configmaps verbs: - get - watch - list - apiGroups: - "" resources: - secrets verbs: - get - create - apiGroups: - "" resources: - pods - pods/exec - pods/log verbs: - get - list - watch - delete - apiGroups: - "" resources: - events verbs: - watch - create - patch - apiGroups: - "" resources: - serviceaccounts verbs: - get - list - watch - apiGroups: - argoproj.io resources: - eventsources - sensors - workflows - workfloweventbindings - workflowtemplates - cronworkflows - clusterworkflowtemplates verbs: - create - get - list - watch - update - patch - delete ```Server ClusterRole
Additional Info
In a Slack convo Anton suggested that https://github.com/argoproj/argo-workflows/pull/12573 may be suspect.
Link to slack message thread
People engaged already:
Happy to provide any more information on request or connect to experiment/work through the issue
Version
v3.5.5
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
Logs from the workflow controller
Logs from in your workflow's wait container