[stable/kong] Connection hangs when cert is enabled on the ELB

sekka1 commented 5 years ago

Is this a request for help?: Yes

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Bug or Im using this wrong

Version of Helm and Kubernetes:

$ helm version
Client: &version.Version{SemVer:"v2.10.0", GitCommit:"9ad53aac42165a5fadc6c87be0dea6b115f93090", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.3", GitCommit:"eecf22f77df5f65c823aacd2dbd30ae6c65f186e", GitTreeState:"clean"}

$ kubectl  version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:25:46Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart: stable/kong

What happened: When trying to set up this topology:

Internet <--> (HTTPS:Valid cert here) AWS ELB <----> (HTTPS) kong <---> (http) pod

The connection from the ELB to Kong never opens and if you cURL this endpoint it just hangs until the ELB times out the connection. I also do not see any log entries in kong for this connection.

As far as I know, the ELB will not do cert validation on the cert that Kong returns.

Not sure if I'm missing any other configs needed?

What you expected to happen: I would expect the ELB to open a connection to kong on the SSL port.

How to reproduce it (as minimally and precisely as possible):

Values yaml file

# Default values for kong.
# Declare variables to be passed into your templates.

image:
  repository: kong
  tag: 1.0.2
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  # pullSecrets:
  #   - myRegistrKeySecretName

waitImage:
  repository: busybox
  tag: latest

# Specify Kong admin and proxy services configurations
admin:
  # If you want to specify annotations for the admin service, uncomment the following
  # line, add additional or adjust as needed, and remove the curly braces after 'annotations:'.
  annotations: {}
  #  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

  # HTTPS traffic on the admin port
  # if set to false also set readinessProbe and livenessProbe httpGet scheme's to 'HTTP'
  useTLS: true
  servicePort: 8444
  containerPort: 8444
  # Kong admin service type
  type: NodePort
  # Set a nodePort which is available
  # nodePort: 32444
  # Kong admin ingress settings.
  ingress:
    # Enable/disable exposure using ingress.
    enabled: false
    # TLS secret name.
    # tls: kong-admin.example.com-tls
    # Array of ingress hosts.
    hosts: []
    # Map of ingress annotations.
    annotations: {}
    # Ingress path.
    path: /

proxy:
  # If you want to specify annotations for the proxy service, uncomment the following
  # line, add additional or adjust as needed, and remove the curly braces after 'annotations:'.
  annotations:
  #  service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "https"
    service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
    external-dns.alpha.kubernetes.io/hostname: foo.bar
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:xxxxxx

  # HTTP plain-text traffic
  http:
    enabled: true
    servicePort: 80
    containerPort: 8000
    # Set a nodePort which is available if service type is NodePort
    # nodePort: 32080

  tls:
    enabled: true
    servicePort: 443
    containerPort: 8443
    # Set a nodePort which is available if service type is NodePort
    # nodePort: 32443

  type: LoadBalancer

  # Kong proxy ingress settings.
  ingress:
    # Enable/disable exposure using ingress.
    enabled: false
    # TLS secret name.
    # tls: kong-proxy.example.com-tls
    # Array of ingress hosts.
    hosts: []
    # Map of ingress annotations.
    annotations: {}
    # Ingress path.
    path: /

# Set runMigrations to run Kong migrations
runMigrations: true

# Specify Kong configurations
# Kong configurations guide https://getkong.org/docs/latest/configuration/
env:
  database: postgres

# If you want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
resources: {}
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

# readinessProbe for Kong pods
readinessProbe:
  httpGet:
    path: "/status"
    port: admin
    scheme: HTTPS
  initialDelaySeconds: 30
  timeoutSeconds: 1
  periodSeconds: 10
  successThreshold: 1
  failureThreshold: 5

# livenessProbe for Kong pods
livenessProbe:
  httpGet:
    path: "/status"
    port: admin
    scheme: HTTPS
  initialDelaySeconds: 30
  timeoutSeconds: 5
  periodSeconds: 30
  successThreshold: 1
  failureThreshold: 5

# Affinity for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
# affinity: {}

# Tolerations for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []

# Node labels for pod assignment
# Ref: https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector: {}

# Annotation to be added to Kong pods
podAnnotations: {}

# Kong pod count
replicaCount: 1

# Kong has a choice of either Postgres or Cassandra as a backend datatstore.
# This chart allows you to choose either of them with the `database.type`
# parameter.  Postgres is chosen by default.

# Additionally, this chart allows you to use your own database or spin up a new
# instance by using the `postgres.enabled` or `cassandra.enabled` parameters.
# Enabling both will create both databases in your cluster, but only one
# will be used by Kong based on the `env.database` parameter.
# Postgres is enabled by default.

# Cassandra chart configs
cassandra:
  enabled: false

# PostgreSQL chart configs
postgresql:
  enabled: true
  postgresqlUsername: kong
  postgresqlDatabase: kong
  service:
    port: 5432

# Kong Ingress Controller's primary purpose is to satisfy Ingress resources
# created in k8s.  It uses CRDs for more fine grained control over routing and
# for Kong specific configuration.
ingressController:
  enabled: true
  image:
    repository: kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller
    tag: 0.3.0
  replicaCount: 1
  livenessProbe:
    failureThreshold: 3
    httpGet:
      path: "/healthz"
      port: 10254
      scheme: HTTP
    initialDelaySeconds: 30
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5
  readinessProbe:
    failureThreshold: 3
    httpGet:
      path: "/healthz"
      port: 10254
      scheme: HTTP
      initialDelaySeconds: 30
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 5

  installCRDs: true

  rbac:
    # Specifies whether RBAC resources should be created
    create: true

  serviceAccount:
    # Specifies whether a ServiceAccount should be created
    create: true
    # The name of the ServiceAccount to use.
    # If not set and create is true, a name is generated using the fullname template
    name:

  ingressClass: kong-gar

Test echo pod

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: http-svc
  name: http-svc
  namespace: gar
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: http-svc
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: http-svc
    spec:
      containers:
      - env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        image: gcr.io/google_containers/echoserver:1.8
        imagePullPolicy: IfNotPresent
        name: http-svc
        ports:
        - containerPort: 8080
          protocol: TCP
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: http-svc
  name: http-svc
  namespace: gar
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: http-svc
  sessionAffinity: None
  type: ClusterIP

Ingress created

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: kong-gar
  name: foo-bar
  namespace: gar
spec:
  rules:
  - host: foo.bar
    http:
      paths:
      - backend:
          serviceName: http-svc
          servicePort: 80
        path: /

Hangs on HTTPS

$ curl -k -v https://internal-a37f7a70124af11e98b230e55eef7289-752875844.us-east-1.elb.amazonaws.com -H "HOST: foo.bar"
* Rebuilt URL to: https://internal-a37f7a70124af11e98b230e55eef7289-752875844.us-east-1.elb.amazonaws.com/
*   Trying 10.4.0.74...
* TCP_NODELAY set
* Connected to internal-a37f7a70124af11e98b230e55eef7289-752875844.us-east-1.elb.amazonaws.com (10.4.0.74) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.dev.q-internal.tech
*  start date: Mar  1 00:00:00 2018 GMT
*  expire date: Apr  1 12:00:00 2019 GMT
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.1
> HOST: foo.bar
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 408 REQUEST_TIMEOUT
< Content-Length:0
< Connection: Close
< 
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

Success on HTTP

$ curl -k -v http://internal-a37f7a70124af11e98b230e55eef7289-752875844.us-east-1.elb.amazonaws.com -H "HOST: foo.bar"
* Rebuilt URL to: http://internal-a37f7a70124af11e98b230e55eef7289-752875844.us-east-1.elb.amazonaws.com/
*   Trying 10.4.0.74...
* TCP_NODELAY set
* Connected to internal-a37f7a70124af11e98b230e55eef7289-752875844.us-east-1.elb.amazonaws.com (10.4.0.74) port 80 (#0)
> GET / HTTP/1.1
> HOST: foo.bar
> User-Agent: curl/7.58.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: text/plain; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Date: Wed, 30 Jan 2019 16:58:17 GMT
< Server: echoserver
< X-Kong-Upstream-Latency: 1
< X-Kong-Proxy-Latency: 14
< Via: kong/1.0.2
< 

Hostname: http-svc-794dc89f5-cjmvj

Pod Information:
    node name:  ip-10-4-6-10.ec2.internal
    pod name:   http-svc-794dc89f5-cjmvj
    pod namespace:  gar
    pod IP: 100.96.5.57

Server values:
    server_version=nginx: 1.13.3 - lua: 10008

Request Information:
    client_address=100.96.5.110
    method=GET
    real path=/
    query=
    request_version=1.1
    request_uri=http://100.96.5.57:8080/

Request Headers:
    accept=*/*
    connection=keep-alive
    host=100.96.5.57:8080
    user-agent=curl/7.58.0
    x-forwarded-for=10.4.6.10
    x-forwarded-host=foo.bar
    x-forwarded-port=8000
    x-forwarded-proto=http
    x-real-ip=10.4.6.10

Request Body:
    -no body in request-

Anything else we need to know: I am unclear on why the http endpoint hangs.

Happy to troubleshoot this if anyone has any ideas.

sekka1 commented 5 years ago

@hbagdi Looking for help with this problem. Thanks.

sekka1 commented 5 years ago

Ping ping @hbagdi. Could use some help with this problem. Can you take a look?

hbagdi commented 5 years ago

@sekka1,

Apologies for the delay, I was out for a while.

Which port on Kong is ELB talking to? It looks like ELB has connectivity issues with Kong, right?

Can you try setting up ELB to talk to Kong over HTTP and test it out? Meaning TLS termination will be performed by the ELB and then ELB <-> Kong will be plaintext.

sekka1 commented 5 years ago

@hbagdi Welcome back =)

Here is the ELB loadbalancer that it create: screenshot from 2019-02-23 08-06-29

Which matches up to the kong-proxy service:

$ kubectl -n gar describe svc gar-kong-kong-proxy
Name:                     gar-kong-kong-proxy
Namespace:                gar
Labels:                   app=kong
                          chart=kong-0.9.2
                          heritage=Tiller
                          release=gar-kong
Annotations:              
                          service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
                          service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
                          service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-east-1:foobar-removed
                          service.beta.kubernetes.io/aws-load-balancer-ssl-ports: 443
Selector:                 app=kong,component=app,release=gar-kong
Type:                     LoadBalancer
IP:                       100.64.202.196
LoadBalancer Ingress:     internal-removed.us-east-1.elb.amazonaws.com
Port:                     kong-proxy  80/TCP
TargetPort:               8000/TCP
NodePort:                 kong-proxy  32693/TCP
Endpoints:                100.96.4.36:8000
Port:                     kong-proxy-tls  443/TCP
TargetPort:               8443/TCP
NodePort:                 kong-proxy-tls  30731/TCP
Endpoints:                100.96.4.36:8443
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Which port on Kong is ELB talking to?

It should be going to port 8443 on kong.

It looks like ELB has connectivity issues with Kong, right?

Yes, it looks like ELB has connectivity issues with Kong. Looks like something is happening thought because if I cURL it the connect hangs for 60 seconds timeout period of ELB before returning instead of just rejecting the connection out right. Not sure but seems like something is happening but cant tell what.

Can you try setting up ELB to talk to Kong over HTTP and test it out? Meaning TLS termination will be performed by the ELB and then ELB <-> Kong will be plaintext.

I have tried this and this and it works. The ELB can terminate the TLS connection and send in clear text to kong and this path works. Dont know if you remember but this was what the TLS offloading PRs I sent a while back did. Dont think we ever got those merged. However, we would like TLS all the way through if possible. Hopefully, you have some more thoughts on this connection or how to troubleshoot it. Happy to try out some of your thoughts on this.

hbagdi commented 5 years ago

Can you make sure that security groups are configured correctly to allow ELB to reach Kong on the TLS port? Also, another user ran into a similar problem recently, so may be trying changing the ciphers that Kong accepts during TLS negotiation (ssl_cipher_suite kong.conf property).

sekka1 commented 5 years ago

The ELB's SG rules look ok. Tried out the ssl_cipher_suite you link to and that was it.

Thanks @hbagdi !!!

hbagdi commented 5 years ago

Glad to hear that was it.

helm / charts