Post /token: x509: certificate signed by unknown authority

[jonbesga]

Hi, I'm having the same problem as #74 but can't get to solve it with the information available there.

My current configMap and deployment are:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gangway
  namespace: auth-system
data:
  gangway.yaml: |
    clusterName: "K8S"
    apiServerURL: "https://10.64.160.231:6443"
    authorizeURL: "https://auth.kubernetes.domain.net/auth"
    tokenURL: "https://auth.kubernetes.domain.net/token"
    clientID: "oidc-auth-client"
    clientSecret: "*******************"
    redirectURL: "https://kubectl.kubernetes.domain.net/callback"
    scopes: ["openid", "profile", "email", "offline_access"]
    usernameClaim: "email"
    emailClaim: "email"
    trustedCAPath: "/cacerts/tls.crt"

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: gangway
  namespace: auth-system
  labels:
    app: gangway
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gangway
  strategy:
  template:
    metadata:
      labels:
        app: gangway
        revision: "1"
    spec:
      containers:
        - name: gangway
          image: gcr.io/heptio-images/gangway:v2.0.0
          imagePullPolicy: Always
          command: ["gangway", "-config", "/gangway/gangway.yaml"]
          env:
            - name: GANGWAY_SESSION_SECURITY_KEY
              valueFrom:
                secretKeyRef:
                  name: gangway-key
                  key: sesssionkey
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
          resources:
            requests:
              cpu: "100m"
              memory: "100Mi"
            limits:
              cpu: "100m"
              memory: "100Mi"
          volumeMounts:
            - name: gangway
              mountPath: /gangway/
            - name: ca
              mountPath: /cacerts/
          livenessProbe:
            httpGet:
              path: /
              port: 8080
            initialDelaySeconds: 20
            timeoutSeconds: 1
            periodSeconds: 60
            failureThreshold: 3
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            timeoutSeconds: 1
            periodSeconds: 10
            failureThreshold: 3
      volumes:
        - name: gangway
          configMap:
            name: gangway
        - name: ca
          secret:
            secretName: rootca

And the secret rootca was creating using:

For the root certificate: openssl genrsa -des3 -out rootCA.key 4096

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.crt

kubectl create secret tls rootCA --key rootCA.key --cert rootCA.crt --namespace auth-system

For creating the certificates I followed the following steps:

For the certificate used when loading gangway:

openssl genrsa -out auth.kubernetes.domain.net.key 2048

openssl req -new -sha256 -key auth.kubernetes.domain.net -subj "/C=US/ST=CA/O=MyOrg, Inc./CN=auth.kubernetes.domain.net" -out auth.kubernetes.domain.net.csr

openssl x509 -req -in auth.kubernetes.domain.net.csr -CA rootCA.crt -CAkey rootCA.key -CAcreateserial -out auth.kubernetes.domain.net.crt -days 500 -sha256

kubectl create secret tls auth.kubernetes.domain.net --key auth.kubernetes.domain.net.key --cert auth.kubernetes.domain.net.crt

Which is used in the following daemonset (I have a nginx daemonset in the nodes so I don't use Ingress):

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
      - image: nginx:1.15.3-alpine
        name: nginx
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
          hostPort: 443
        volumeMounts:
        - name: "config"
          mountPath: "/etc/nginx"
        - name: secret-volume
          mountPath: /etc/nginx/ssl
      volumes:
      - name: config
        configMap:
          name: nginx-conf
      - name: secret-volume
        secret:
           secretName: auth.kubernetes.domain.net

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-conf
data:
  nginx.conf: |
    worker_processes 1;
    error_log /dev/stdout info;

    events {
      worker_connections 10;
    }

    http {
      access_log /dev/stdout;

      server {
        listen 443 ssl;
        server_name auth.kubernetes.domain.net;
        ssl_certificate /etc/nginx/ssl/tls.crt;
        ssl_certificate_key /etc/nginx/ssl/tls.key;

        location / {
          proxy_pass http://dex.auth-system.svc.cluster.local:5556;
        }
      }
      server {
        server_name kubectl.kubernetes.domain.net
        ssl_certificate /etc/nginx/ssl/tls.crt;
        ssl_certificate_key /etc/nginx/ssl/tls.key;
        listen 443 ssl;
        #listen 80;
        location / {
          proxy_pass http://gangway-svc.auth-system.svc.cluster.local:80;
        }
      }
   }

[alexbrand]

@jonbesga Are there any logs coming out of Gangway?

[jonbesga]

Thanks for the help @alexbrand. I don't see anything strange.

time="2019-03-12T14:26:22Z" level=info msg="GET / 10.64.160.232:46268"
time="2019-03-12T14:26:22Z" level=info msg="GET /js/materialize.js 10.64.160.232:46270"
time="2019-03-12T14:26:22Z" level=info msg="GET /js/init.js 10.64.160.232:46272"
time="2019-03-12T14:26:23Z" level=info msg="GET /favicon.ico 10.64.160.232:46274"
time="2019-03-12T14:26:27Z" level=info msg="GET /login 10.64.160.232:46280"
time="2019-03-12T14:26:30Z" level=info msg="GET / 10.64.160.232:56610"
time="2019-03-12T14:26:32Z" level=info msg="GET /callback?code=y5uije2rjb3wvest2wmyo57ho&state=gYVaGx1J1JVchIYhYyUlP%2Bxzjdep4ov5IRGcFg8HAkQ%3D 10.64.160.232:46298"
time="2019-03-12T14:26:32Z" level=info msg="GET /favicon.ico 10.64.160.232:46302"

[alexbrand]

At quick glance, it all looks good to me from a configuration perspective.

It might be worth verifying that the certs line up with our expectations.

1. Download the cert being presented by the remote server. Something like this should do it:

   openssl s_client -showcerts -connect auth.kubernetes.domain.net:443 < /dev/null 2> /dev/null | openssl x509 -outform PEM >auth.kubernetes.domain.net.pem

2. Download the CA cert that gangway is using:

   kubectl -n auth-system cp ${gangwayPod}:/cacerts/tls.crt gangway-ca.crt

3. Verify the server's cert with the CA used by gangway:

   openssl verify -CAfile gangway-ca.crt auth.kubernetes.domain.net.pem

[jonbesga]

I had to get inside the gangway container to read the /cacerts/tls.crt (seems that I can't cp a file mounted from secrets)

The third step is giving me an OK:

$ openssl verify -CAfile gangway-ca.crt auth.kubernetes.domain.net.pem
auth.kubernetes.domain.net.pem: OK

[alexbrand]

Looks like you are using v2.0.0. The trustedCAPath option was introduced in a later version. Can you try updating to the latest version (v3.0.0)?

[jonbesga]

Brilliant! It worked! Thank you very much.

[alexbrand]

Excellent! I'll go ahead and close this issue.

[kotnn]

I use the latest version.

[kotnn]

The issue was resolved by re-creating docker image with: 1) git clone https://github.com/heptiolabs/gangway 2) cd gangway 3) cp -p Dockerfile Dockerfile.orig 4) vi ca.pem Content of file: -----BEGIN CERTIFICATE----- certificate chain -----END CERTIFICATE----- 5) change Dockerfile content to

FROM golang:1.14.2-stretch WORKDIR /go/src/github.com/heptiolabs/gangway

RUN go get -u github.com/mjibson/esc/... COPY . . RUN esc -o cmd/gangway/bindata.go templates/

ENV GO111MODULE on RUN go mod verify RUN CGO_ENABLED=0 GOOS=linux go install -ldflags="-w -s" -v github.com/heptiolabs/gangway/...

FROM debian:9.12-slim COPY ./ca.pem /usr/local/share/ca-certificates/myca.pem RUN apt-get update && apt-get install -y ca-certificates USER 1001:1001 COPY --from=0 /go/bin/gangway /bin/gangway

6)build new docker image from this Dockerfile