search

dexidp / dex

OpenID Connect (OIDC) identity and OAuth 2.0 provider with pluggable connectors
https://dexidp.io
Apache License 2.0
9.5k stars 1.71k forks source link

Oidc authenticator: initializing plugin: Get https://172.16.x.x/.well-known/openid-configuration: Forbidden #2171

Closed brokencode64 closed 3 years ago

brokencode64 commented 3 years ago

Summary

We are using Gangway/dex to authenticate to our ldap servers. I created a similar setup in aws and it worked perfectly.

Expected behavior (what you expected to happen)

Kubectl should be able to get resources with downloaded config.

Actual behavior (what actually happened)

We are able to login to gangway/dex and get a kubeconfig to download. Dex is clearly getting ldap groups etc, but when you try to use the kubeconfig you get this error:

Kubectl commands result in the error:

error: You must be logged in to the server (Unauthorized)

Error Logs

Kubectl logs:

error: You must be logged in to the server (Unauthorized)

Dex error logs (In debug mode)

http: TLS handshake error from 100.x.x.x:x: local error: tls: bad record MAC
http: TLS handshake error from 172.x.x.x:x: local error: tls: bad record MAC

API server error logs:

Unable to authenticate the request due to an error: [invalid bearer token, oidc: authenticator not initialized]
. . .

 oidc authenticator: initializing plugin: Get https://172.x.x.x/.well-known/openid-configuration: Forbidden

Environment

Tested

  1. Double-triple checked the certs and they all look good
  2. Decoded my JWT token and it looks good
  3. API server can reach the endpoint (I can curl it even with --insecure)
  4. I can reach API server and dex endpoint with curl

Manifests

Apiserver configuration:

spec:                                                                                                                                                                                                                                                                                                                          
  containers:                                                                                                                                                                                                                                                                                                                  
  - command:                                                                                                                                                                                                                                                                                                                   
    - kube-apiserver                                                                                                                                                                                                                                                                                                           
    - --advertise-address=100.106.x.x                                                                                                                                                                                                                                                                                        
    - --allow-privileged=true                                                                                                                                                                                                                                                                                                  
    - --anonymous-auth=true                                                                                                                                                                                                                                                                                                    
    - --apiserver-count=3                                                                                                                                                                                                                                                                                                      
    - --oidc-issuer-url=https://172.16.x.x                                                                                                                                                                                                                                                                                  
    - --oidc-client-id=gangway-lab-test                                                                                                                                                                                                                                                                                         
    - --oidc-username-claim=name                                                                                                                                                                                                                                                                                               
    - --oidc-groups-claim=groups                                                                                                                                                                                                                                                                                               
    - --oidc-ca-file=/etc/kubernetes/ssl/oidc-ca.pem                                                                                                                                                                                                                                                                           
    - '--oidc-groups-prefix=oidc:'                                                                                                                                                                                                                                                                                             
    - --authorization-mode=Node,RBAC                                                                                                                                                                                                                                                                                           
    - --basic-auth-file=/etc/kubernetes/users/known_users.csv                                                                                                                                                                                                                                                                  
    - --bind-address=0.0.0.0                                                                                                                                                                                                                                                                                                   
    - --client-ca-file=/etc/kubernetes/ssl/ca.crt                                                                                                                                                                                                                                                                              
    - --enable-admission-plugins=NodeRestriction                                                                                                                                                                                                                        
    - --enable-aggregator-routing=False                                                                                                                                                                                                                                                                                        
    - --enable-bootstrap-token-auth=true                                                                                                                                                                                                                                                                                       
    - --endpoint-reconciler-type=lease                                                                                                                                                                                                                                                                                         
    - --etcd-cafile=/etc/ssl/etcd/ssl/ca.pem                                                                                                                                                                                                                                                                                   
    - --etcd-certfile=/etc/ssl/etcd/ssl/node-x.pem                                                                                                                                                                                                                                                               
    - --etcd-keyfile=/etc/ssl/etcd/ssl/node-x.pem                                                                                                                                                                                                                                                            
    - --etcd-servers=https://100.106.x.x:2379,https://100.106.x.x:2379,https://100.106.x.x:2379                                                                                                                                                                                                                          
    - --event-ttl=1h0m0s                                                                                                                                                                                                                                                                                                       
    - --insecure-port=0                                                                                                                                                                                                                                                                                                        
    - --kubelet-client-certificate=/etc/kubernetes/ssl/apiserver-kubelet-client.crt                                                                                                                                                                                                                                            
    - --kubelet-client-key=/etc/kubernetes/ssl/apiserver-kubelet-client.key                                                                                                                                                                                                                                                    
    - --kubelet-preferred-address-types=InternalDNS,InternalIP,Hostname,ExternalDNS,ExternalIP                                                                                                                                                                                                                                 
    - --profiling=False                                                                                                                                                                                                                                                                                                        
    - --proxy-client-cert-file=/etc/kubernetes/ssl/front-proxy-client.crt                                                                                                                                                                                                                                                      
    - --proxy-client-key-file=/etc/kubernetes/ssl/front-proxy-client.key                                                                                                                                                                                                                                                       
    - --request-timeout=1m0s                                                                                                                                                                                                                                                                                                   
    - --requestheader-allowed-names=front-proxy-client                                                                                                                                                                                                                                                                         
    - --requestheader-client-ca-file=/etc/kubernetes/ssl/front-proxy-ca.crt                                                                                                                                                                                                                                                    
    - --requestheader-extra-headers-prefix=X-Remote-Extra-                                                                                                                                                                                                                                                                     
    - --requestheader-group-headers=X-Remote-Group                                                                                                                                                                                                                                                                             
    - --requestheader-username-headers=X-Remote-User                                                                                                                                                                                                                                                                           
    - --secure-port=6443                                                                                                                                                                                                                                                                                                       
    - --service-account-key-file=/etc/kubernetes/ssl/sa.pub                                                                                                                                                                                                                                                                    
    - --service-cluster-ip-range=100.80.0.0/12                                                                                                                                                                                                                                                                                 
    - --service-node-port-range=30000-32767                                                                                                                                                                                                                                                                                    
    - --storage-backend=etcd3                                                                                                                                                                                                                                                                                                  
    - --tls-cert-file=/etc/kubernetes/ssl/apiserver.crt                                                                                                                                                                                                                                                                        
    - --tls-private-key-file=/etc/kubernetes/ssl/apiserver.key                                                                                                                                                                                                                                                                 
    - --token-auth-file=/etc/kubernetes/tokens/known_tokens.csv 
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: dex
  namespace: kube-system
data:
  config.yaml: |
    issuer: https://172.x.x.x
    storage:
      type: kubernetes
      config:
        inCluster: true
    logger:
      level: debug
    web:
      https: 0.0.0.0:5556
      tlsCert: /etc/dex/tls/tls.crt
      tlsKey: /etc/dex/tls/tls.key
    connectors:
    - type: ldap
      name: LDAP
      id: ldap
      config:
        host: 172.x.x.x
        insecureNoSSL: true
        bindDN: uid=test,ou=example,dc=svc,dc=com
        bindPW: ${BIND_PWD}
        usernamePrompt: SSO Username
        userSearch:
          baseDN: ou=example,dc=svc,dc=com
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: uid
        groupSearch:
          baseDN: ou=example,ou=svc,dc=com
          filter: "(objectClass=posixGroup)"
          userMatchers:
          - userAttr: DN
            groupAttr: member
            nameAttr: name
          nameAttr: cn
    oauth2:
      skipApprovalScreen: true
    staticClients:
    - id: gangway-lab-test
      redirectURIs:
      - 'http://gangway.example.com/callback'
      name: 'gangway-lab-test'
      secret: supersecret                        
    expiry:
     signingKeys: "6h"
     idTokens: "24h"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: dex
  name: dex
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dex
  template:
    metadata:
      labels:
        app: dex
    spec:
      serviceAccountName: dex
      containers:
      - image: quay.io/dexidp/dex:v2.28.0
        name: dex
        command: ["/usr/local/bin/dex", "serve", "/etc/dex/cfg/config.yaml"]
        ports:
        - name: https
          containerPort: 5556
        volumeMounts:
        - name: config
          mountPath: /etc/dex/cfg
        - name: tls
          mountPath: /etc/dex/tls
        env:
        - name: BIND_PWD
          valueFrom:
            secretKeyRef:
              name: dex-credentials
              key: bind.pwd
        - name: GANGWAY_SECRET
          valueFrom:
            secretKeyRef:
              name: dex-credentials
              key: gangway.secret
        envFrom:
      volumes:
      - name: config
        configMap:
          name: dex
          items:
          - key: config.yaml
            path: config.yaml
      - name: tls
        secret:
          secretName: dex-tls
brokencode64 commented 3 years ago

Issue was an http proxy between api-server and dex. Had to add dex ip to "No-proxy" in kube-apiserver settings.