search

Kong / charts

Helm chart for Kong
Apache License 2.0
248 stars 480 forks source link

TLS with user provided secret doesn't seem to work #932

Open pmalek opened 12 months ago

pmalek commented 12 months ago

Problem statement

While trying to make mTLS between KIC and Admin API work I encountered some difficulties so I thought I'd share some of them.

Using the following script to generate certs and create Kubernetes Secrets:

#!/usr/bin/env bash

set -o pipefail
set -e
set -x

CACERT=ca.crt
CAKEY=ca.key
TLSKEY=tls.key
TLSCERT=tls.crt
TLSCSR=tls.csr

echo "Creating root CA" && \
  openssl \
  req -x509 -sha256 -days 365 -newkey rsa:2048 -nodes \
  -subj "/C=US/ST=County/L=City/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod.cluster.local" \
  -keyout ${CAKEY} -out ${CACERT}

openssl req -newkey rsa:2048 -keyout ${TLSKEY} -nodes \
  -subj "/C=US/ST=County/L=City/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod.cluster.local" \
  -out ${TLSCSR}

openssl x509 -req -CA ${CACERT} -CAkey ${CAKEY} -in ${TLSCSR} -out ${TLSCERT} \
  -subj "/C=US/ST=County/L=City/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod.cluster.local" \
  -days 365 -CAcreateserial -extfile domain.ext

(
cat << EOF
apiVersion: v1
kind: Secret
metadata:
  name: adminapi-cert
  namespace: kong
type: kubernetes.io/tls
data:
  tls.crt: $(base64 -i ${TLSCERT})
  tls.key: $(base64 -i ${TLSKEY})
EOF
) | kubectl apply -f -

(
cat << EOF
apiVersion: v1
kind: Secret
metadata:
  name: ca-cert
  namespace: kong
type: kubernetes.io/tls
data:
  tls.crt: $(base64 -i ${CACERT})
  tls.key: $(base64 -i ${CAKEY})
EOF
) | kubectl apply -f -

domain.ext file

authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.kong.pod
DNS.2 = *.kong.pod.cluster.local
DNS.3 = *.kong-gateway-admin.kong.svc

and these values.yaml for the ingress chart:

gateway:
  replicaCount: 2
  readinessProbe:
    initialDelaySeconds: 0
    timeoutSeconds: 1
    periodSeconds: 1

controller:
  ingressController:
    image:
      repository: kong/kubernetes-ingress-controller
      tag: "3.0"

    adminApi:
      tls:
        client:
          enabled: true
          # If set to true, you are expected to provide your own secret (see secretName, caSecretName).
          certProvided: true
          # Client TLS certificate/key pair secret name that Ingress Controller will use to authenticate with Kong Admin API.
          # If certProvided is set to false, it is optional (can be specified though if you want to force Helm to use
          # a specific secret name).
          secretName: "adminapi-cert"
          # CA TLS certificate/key pair secret name that the client TLS certificate is signed by.
          # If certProvided is set to false, it is optional (can be specified though if you want to force Helm to use
          # a specific secret name).
          caSecretName: "ca-cert"

    env:
      gateway_discovery_dns_strategy: pod
      log_level: debug
      anonymous_reports: "false"
      kong_admin_tls_skip_verify: "false"

      ## Do we need this?
      # kong_admin_tls_server_name: ""
      kong_admin_ca_cert:
        valueFrom:
          secretKeyRef:
             name: ca-cert
             key: tls.crt

I keep getting 

2023-11-06T19:50:58Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "0/60", "error": "making HTTP request: Get \"https://10-244-0-87.kong.pod:8444/\": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match 10-244-0-87.kong.pod"}

The certificates I'm able to get from the Admin API seem to be OK:

k get secret -n kong adminapi-cert -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text | less
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            59:d1:b8:63:5a:46:76:8c:2c:13:0d:bb:aa:01:cd:ae:78:f6:99:7c
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = PL, ST = County, L = City, O = Kong, OU = Kong_Kubernetes, CN = *.kong.pod.cluster.local
        Validity
            Not Before: Nov  6 19:27:10 2023 GMT
            Not After : Nov  5 19:27:10 2024 GMT
        Subject: C = PL, ST = County, L = City, O = Kong, OU = Kong_Kubernetes, CN = *.kong.pod.cluster.local
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:cc:fe:aa:c3:d8:6f:f8:6d:64:a3:ad:5b:62:42:
                    0f:a4:45:d6:37:df:60:10:34:dc:3b:1f:66:e4:5c:
                    12:47:cf:4a:cd:94:21:96:81:51:6a:6b:20:63:08:
                    34:71:2f:29:9a:69:cf:e4:80:10:0f:a3:1b:c4:d4:
                    36:c3:b7:45:d6:6e:a3:50:93:ac:c4:d5:49:16:ab:
                    a3:28:40:c2:ab:bd:9b:6f:9d:25:f1:3f:7e:fd:13:
                    a7:0c:c9:30:ae:2a:1d:42:3b:23:dd:03:dd:df:0f:
                    12:26:60:df:49:ec:24:e0:3f:49:38:f3:0c:0a:c2:
                    47:cf:35:d6:65:36:84:8d:e4:e5:e6:55:ec:41:34:
                    10:d4:40:06:c2:1e:38:1c:28:5c:b3:93:69:49:a8:
                    05:22:9f:f5:de:3e:0b:f5:a4:0c:f0:13:e2:18:67:
                    45:40:df:21:8a:08:c9:e3:2e:05:94:0e:4b:0d:85:
                    fd:fe:a6:96:9e:fb:12:59:d1:fc:e6:73:89:0f:65:
                    07:de:5d:b9:39:d2:fa:77:06:bb:8f:25:42:af:ce:
                    78:7d:08:ad:dd:a0:19:ca:15:6d:5b:d1:84:2f:0a:
                    ec:a2:85:67:aa:25:6a:ab:7c:8b:e6:ff:56:f0:53:
                    dd:2a:70:7f:7b:85:fe:16:06:80:99:e8:ae:17:38:
                    df:eb
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Authority Key Identifier: 
                C9:2E:D0:77:1B:FB:63:71:02:7F:1E:A8:C3:5F:BB:8C:2D:85:A2:4E
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:*.kong.pod, DNS:*.kong.pod.cluster.local, DNS:*.kong-gateway-admin.kong.svc
            X509v3 Subject Key Identifier: 
                62:CA:00:D1:E8:48:51:7B:B7:DF:17:8D:EB:3E:09:C4:89:A1:01:58
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        72:33:09:24:92:2b:12:a2:0c:e3:5a:f7:e3:a7:f2:c5:e5:4e:
        3d:51:29:68:16:b6:12:fc:7a:a4:33:58:b4:60:80:30:52:dd:
        e3:38:d3:b1:28:45:29:86:c9:53:f4:a8:c8:ca:9d:87:c8:77:
        ee:1d:ea:c8:1f:81:aa:87:6d:ec:16:cd:be:08:b8:ad:1a:a9:
        24:7f:9b:b9:b3:c1:44:7f:a7:4e:91:5b:b9:2b:1e:f4:28:3e:
        f8:a8:8e:14:5f:09:37:9b:e8:cf:40:e7:b7:1f:88:25:d2:75:
        17:cf:4b:3a:ad:58:1f:71:62:db:c9:60:10:33:13:b4:b9:6a:
        7c:74:b5:68:14:73:89:0e:70:c1:33:0b:d5:f9:07:a1:11:7b:
        43:a8:58:55:21:c2:a6:3c:df:40:5d:14:88:f8:b3:30:6a:59:
        0b:2c:c3:d7:85:e2:81:4f:fd:e6:f6:7b:3e:8f:64:72:04:c5:
        c6:80:18:96:b5:20:c2:9f:71:a6:a5:47:1d:0a:b5:4d:c5:e9:
        b7:f9:25:87:58:e0:9a:f6:9c:0a:8a:58:2e:6d:19:32:94:05:
        de:e9:6b:a8:88:dd:92:fe:9f:95:b9:ab:1b:e1:b4:eb:ad:df:
        62:33:4f:e4:d0:df:5c:67:76:29:33:4f:38:fd:5e:49:93:f0:
        10:d6:52:8e
-----BEGIN CERTIFICATE-----
MIIEHzCCAwegAwIBAgIUWdG4Y1pGdowsEw27qgHNrnj2mXwwDQYJKoZIhvcNAQEL
BQAweTELMAkGA1UEBhMCUEwxDzANBgNVBAgMBkNvdW50eTENMAsGA1UEBwwEQ2l0
eTENMAsGA1UECgwES29uZzEYMBYGA1UECwwPS29uZ19LdWJlcm5ldGVzMSEwHwYD
VQQDDBgqLmtvbmcucG9kLmNsdXN0ZXIubG9jYWwwHhcNMjMxMTA2MTkyNzEwWhcN
MjQxMTA1MTkyNzEwWjB5MQswCQYDVQQGEwJQTDEPMA0GA1UECAwGQ291bnR5MQ0w
CwYDVQQHDARDaXR5MQ0wCwYDVQQKDARLb25nMRgwFgYDVQQLDA9Lb25nX0t1YmVy
bmV0ZXMxITAfBgNVBAMMGCoua29uZy5wb2QuY2x1c3Rlci5sb2NhbDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAMz+qsPYb/htZKOtW2JCD6RF1jffYBA0
3DsfZuRcEkfPSs2UIZaBUWprIGMINHEvKZppz+SAEA+jG8TUNsO3RdZuo1CTrMTV
SRaroyhAwqu9m2+dJfE/fv0TpwzJMK4qHUI7I90D3d8PEiZg30nsJOA/STjzDArC
R8811mU2hI3k5eZV7EE0ENRABsIeOBwoXLOTaUmoBSKf9d4+C/WkDPAT4hhnRUDf
IYoIyeMuBZQOSw2F/f6mlp77ElnR/OZziQ9lB95duTnS+ncGu48lQq/OeH0Ird2g
GcoVbVvRhC8K7KKFZ6olaqt8i+b/VvBT3Spwf3uF/hYGgJnorhc43+sCAwEAAaOB
njCBmzAfBgNVHSMEGDAWgBTJLtB3G/tjcQJ/HqjDX7uMLYWiTjAJBgNVHRMEAjAA
ME4GA1UdEQRHMEWCCioua29uZy5wb2SCGCoua29uZy5wb2QuY2x1c3Rlci5sb2Nh
bIIdKi5rb25nLWdhdGV3YXktYWRtaW4ua29uZy5zdmMwHQYDVR0OBBYEFGLKANHo
SFF7t98Xjes+CcSJoQFYMA0GCSqGSIb3DQEBCwUAA4IBAQByMwkkkisSogzjWvfj
p/LF5U49USloFrYS/HqkM1i0YIAwUt3jONOxKEUphslT9KjIyp2HyHfuHerIH4Gq
h23sFs2+CLitGqkkf5u5s8FEf6dOkVu5Kx70KD74qI4UXwk3m+jPQOe3H4gl0nUX
z0s6rVgfcWLbyWAQMxO0uWp8dLVoFHOJDnDBMwvV+QehEXtDqFhVIcKmPN9AXRSI
+LMwalkLLMPXheKBT/3m9ns+j2RyBMXGgBiWtSDCn3GmpUcdCrVNxem3+SWHWOCa
9pwKilgubRkylAXe6WuoiN2S/p+Vuasb4bTrrd9iM0/k0N9cZ3YpM084/V5Jk/AQ
1lKO
-----END CERTIFICATE-----

Trying to access port forwarded Admin API yields:

openssl s_client -showcerts -connect localhost:8443  </dev/null
CONNECTED(00000006)
Can't use SSL_get_servername
depth=0 C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = localhost
verify error:num=18:self-signed certificate
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = localhost
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = localhost
   i:C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = localhost
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Nov  6 19:27:22 2023 GMT; NotAfter: Nov  1 19:27:22 2043 GMT
-----BEGIN CERTIFICATE-----
MIICODCCAd2gAwIBAgIRAIItPBY9Yib9NQTnk7hI+2gwCgYIKoZIzj0EAwIwdTEL
MAkGA1UEBhMCVVMxEzARBgNVBAgMCkNhbGlmb3JuaWExFjAUBgNVBAcMDVNhbiBG
cmFuY2lzY28xDTALBgNVBAoMBEtvbmcxFjAUBgNVBAsMDUlUIERlcGFydG1lbnQx
EjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yMzExMDYxOTI3MjJaFw00MzExMDExOTI3
MjJaMHUxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQH
DA1TYW4gRnJhbmNpc2NvMQ0wCwYDVQQKDARLb25nMRYwFAYDVQQLDA1JVCBEZXBh
cnRtZW50MRIwEAYDVQQDDAlsb2NhbGhvc3QwWTATBgcqhkjOPQIBBggqhkjOPQMB
BwNCAAS/+jrg1VzawnnvB6X5Kovm9Ko3XqcUZY0IsLjKmdRFsDNGAuodn56sP/U+
4NUISjdSCEPlyHgbX9ugMFJferI5o04wTDAMBgNVHRMBAf8EAjAAMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQUgQQzAgwonXBBtjyCVqEu
0I6WI98wCgYIKoZIzj0EAwIDSQAwRgIhAMF4E5n1/rkhOQRThnn2uwlxNvzmPX6e
2MgXX0KpmpNzAiEAn7ARBPKlhvHza+8m/luo3VJUyCXasQze76TlQ098n/I=
-----END CERTIFICATE-----
---
Server certificate
subject=C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = localhost
issuer=C = US, ST = California, L = San Francisco, O = Kong, OU = IT Department, CN = localhost
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 942 bytes and written 373 bytes
Verification error: self-signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self-signed certificate)
---
DONE

(I've tried use different hostname to access it via /etc/hosts but with the same result)

rainest commented 12 months ago

Simple answer here is that the Kong config doesn't have the cert set up. It needs something roughly along the lines of https://github.com/Kong/charts/blob/main/charts/kong/README.md#certificates but for gateway.env.admin_ssl_cert and gateway.env.admin_ssl_cert_key.

That sets up the server certificate serving on the gateway, but not verification of the client certificate presented by the controller. IIRC that relies on injecting NGINX directives (similar to step 8 in https://docs.nginx.com/nginx-management-suite/admin-guides/configuration/secure-traffic/, the ssl_client_certificate and ssl_verify_client bits, so gateway.env.nginx_admin_ssl_client_certificate pointed to the mounted CA cert and gateway.env.nginx_admin_ssl_verify_client=on). I don't think there was ever any dedicated kong.conf functionality for it, and the community addition of the controller client certificate args just used that.

Did you have a doc you were working from? I forget what we created for the rollout of the ingress chart, and it doesn't look like there's anything in the chart README.md for it. We probably want to just inject it automatically in the long run, but can't really with an umbrella chart--one or the other won't be able to reference generated resources easily.

In classical very backwards fashion I pulled this out by decrypting the comms between the two, which was probably unnecessary (coulda just hit the admin API with curl to see the cert details), but I wanted to confirm how to do it. Golang has no runtime built-in key log dump, and you need to modify code:

ingress-controller-key-dump.diff.txt

go-kong-dump.diff.txt

The controller change is the one actually needed, since go-kong lets you provide your own client and we do. Left as a diff archive since I did a hack job and it's one of those things where we probably shouldn't ship it without a global --enforce-production to just exit immediately if anything like that is on.

From there you can dump traffic and kubectl cp the key dumps for Wireshark's TLS decrypt.

pmalek commented 12 months ago

Thanks for pointing me to these envs!

I managed to make it works with the following values:

gateway:
  admin:
    tls:
      client:
        secretName: "ca-cert"
  replicaCount: 2
  env:
    admin_ssl_cert: /etc/secrets/adminapi-cert/tls.crt
    admin_ssl_cert_key: /etc/secrets/adminapi-cert/tls.key
  secretVolumes:
  - adminapi-cert

controller:
  ingressController:
    image:
      repository: kong/kubernetes-ingress-controller
      tag: "3.0"

    adminApi:
      tls:
        client:
          enabled: true
          # If set to true, you are expected to provide your own secret (see secretName, caSecretName).
          certProvided: true
          # Client TLS certificate/key pair secret name that Ingress Controller will use to authenticate with Kong Admin API.
          secretName: "client-adminapi-cert"
          # CA TLS certificate/key pair secret name that the client TLS certificate is signed by.
          caSecretName: "ca-cert"

    env:
      gateway_discovery_dns_strategy: pod
      log_level: debug
      anonymous_reports: "false"
      kong_admin_tls_skip_verify: "false"

      kong_admin_ca_cert:
        valueFrom:
          secretKeyRef:
             name: ca-cert
             key: tls.crt

and the slightly modified script for the certs:

#!/usr/bin/env bash

set -ex -o pipefail

readonly CACERT=ca.crt
readonly CAKEY=ca.key
readonly TLSKEY=tls.key
readonly TLSCERT=tls.crt
readonly TLSCSR=tls.csr
readonly CLIENTTLSKEY=clienttls.key
readonly CLIENTTLSCERT=clienttls.crt
readonly CLIENTTLSCSR=clienttls.csr

readonly DAYS=365
readonly BITS=4096
readonly NAMESPACE=kong

test -f ${CACERT} || (echo "Creating root CA" && \
  openssl \
  req -x509 -sha256 -days ${DAYS} -newkey rsa:${BITS} -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=ca.kong.pod" \
  -keyout ${CAKEY} -out ${CACERT})

openssl req -newkey rsa:${BITS} -keyout ${TLSKEY} -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod" \
  -out ${TLSCSR}

openssl x509 -req -CA ${CACERT} -CAkey ${CAKEY} -in ${TLSCSR} -out ${TLSCERT} \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=*.kong.pod" \
  -days ${DAYS} -CAcreateserial -extfile domain.ext

openssl req -newkey rsa:${BITS} -keyout ${CLIENTTLSKEY} -nodes \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=client.kong.pod" \
  -out ${CLIENTTLSCSR}

openssl x509 -req -CA ${CACERT} -CAkey ${CAKEY} -in ${CLIENTTLSCSR} -out ${CLIENTTLSCERT} \
  -subj "/C=US/ST=California/L=San Francisco/O=Kong/OU=Kong_Kubernetes/CN=client.kong.pod" \
  -days ${DAYS} -CAcreateserial

function secret()
{
  local NAME=${1}
  local NS=${2}
  local CERT=${3}
  local KEY=${4}
kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: "${NAME}"
  namespace: ${NS}
type: kubernetes.io/tls
data:
  tls.crt: $(base64 -i ${CERT})
  tls.key: $(base64 -i ${KEY})
EOF
}

secret client-adminapi-cert ${NAMESPACE} ${CLIENTTLSCERT} ${CLIENTTLSKEY}
secret adminapi-cert ${NAMESPACE} ${TLSCERT} ${TLSKEY}
secret ca-cert ${NAMESPACE} ${CACERT} ${CAKEY}

(domain.ext stays as is)

2023-11-07T18:48:22Z    debug   events  successfully applied Kong configuration to https://10-244-0-200.kong.pod:8444   {"v": 1, "type": "Normal", "object": {"kind":"Pod","namespace":"kong","name":"kong-controller-7cb994b99f-xpxd8","apiVersion":"v1"}, "reason": "KongConfigurationSucceeded"}
pmalek commented 12 months ago

Given the above I wonder if we should close this or consider using this content in a guide in the docs or elsewhere 🤔 ? We don't seem to have a comprehensive guide saying how to configure this. We mostly cover the case of getting and configuring the cert at the ingress level.

rainest commented 12 months ago

Yeah, this basically becomes a docs ticket.

Per Michael we originally had this in https://docs.konghq.com/kubernetes-ingress-controller/2.12.x/guides/using-gateway-discovery/#installation but it hadn't gotten ported yet.