Gateway TLS does not serve intermediate cert chain

[markeijsermans]

Describe the bug

I'm configuring TLS for a gateway in SIMPLE mode. It's a wildcard DNS and the cert is issued by GoDaddy which requires 3 intermediate chained certs. Configuring the gateway with a caCertificates file has no effect.

With scant documentation on this, I attempted configuring certs similar to nginx: http://nginx.org/en/docs/http/configuring_https_servers.html#chains

Expected behavior

The gateway should offer up intermediate certs. Chrome/firefox are ok without the intermediate certs, but most everything else needs it and fails (eg. curl). Non-TLS port 80 is ok too.

Steps to reproduce the bug

The gateway & virtualService definitions are below. I can confirm that the TLS secrets/files are mounted and readable in the gateway pod. The caCertificates is a sinlge bundled cert file in the same format as nginx requires.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: public
spec:
  selector:
    istio: gatewaypublic
  servers:
  - hosts:
    - '*.hootops.com'
    port:
      name: http2
      number: 80
      protocol: HTTP2
  - hosts:
    - '*.hootops.com'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/cert-wildcard-hootops-com/tls.key
      serverCertificate: /etc/istio/cert-wildcard-hootops-com/tls.crt
      caCertificates: /etc/istio/cert-wildcard-hootops-com/ca.crt

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: foo-hootops-com-public
spec:
  gateways:
  - public
  hosts:
  - foo.hootops.com
  http:
  - route:
    - destination:
        host: foo.default.svc.cluster.local
        port:
          number: 8080

When port forwarded into the gateway pod, I see the TLS handshake via openssl client. No certs defined in the caCertificates file is offered up. It seems that parameter is ignored in TLS SIMPLE mode

openssl s_client -showcerts -servername foo.hootops.com -connect localhost:443

CONNECTED(00000003)
depth=0 /OU=Domain Control Validated/CN=*.hootops.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /OU=Domain Control Validated/CN=*.hootops.com
verify error:num=27:certificate not trusted
verify return:1
depth=0 /OU=Domain Control Validated/CN=*.hootops.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/OU=Domain Control Validated/CN=*.hootops.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2
---
Server certificate
-----BEGIN CERTIFICATE-----
...

Version istioctl version

Version: 1.0.2
GitRevision: d639408fded355fb906ef2a1f9e8ffddc24c3d64
User: root@66ce69d4a51e
Hub: gcr.io/istio-release
GolangVersion: go1.10.1
BuildStatus: Clean

kubectl version

Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.3", GitCommit:"2bba0127d85d5a46ab4b778548be28623b32d0b0", GitTreeState:"clean", BuildDate:"2018-05-21T09:05:37Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Installation custom install / minikube

Environment AWS, ubuntu / minikube

Cluster state on request

[markeijsermans]

Error is on my part. Adding the full cert chain to serverCertificate (tls.crt) works. You can close this issue. I'll submit a PR to add clarification to the docs.

[sreenivasmrpivot]

@markeijsermans I have enabled my cluster with self signed cert. The problem I have with it is, "Citadel signs it with its own root/chain". As a result of this, browsers are not able to identify my test1.example.com, test2.example.com etc as secured sites and prompts for exception.

I assume, to avoid this, I will have to plugin external CA as mentioned here. https://istio.io/docs/tasks/security/plugin-ca-cert/

I have my domain in GoDaddy. If I buy my wildcard cert *.example.com from GoDaddy, what files will they give me for download? Where can I get the root cert and cert chain of GoDaddy which needs to be configured on Citadel?

Can I completely avoid Citadel configuration and use the chain in the tls: caCertificates:

Any help is much appreciated.

[manjuprabhuv]

I had the same issue. We use AWS ACM as our certificate manager and set the certificates as kubernetes secrets. I had to modify our script to export both Certificate and Certificate chain in tls.crt. Export the certificate -

 aws acm export-certificate --certificate-arn ${CERTARN} --passphrase ${TLS_SERVER_PROFILE_PASSPHRASE} --cli-binary-format raw-in-base64-out | jq -r '"\(.PrivateKey)"' > tls.key.encrypted
    # istio certificate chain fix - https://github.com/istio/istio/issues/10001
    aws acm export-certificate --certificate-arn ${CERTARN} --passphrase ${TLS_SERVER_PROFILE_PASSPHRASE} --cli-binary-format raw-in-base64-out | jq -r '.Certificate, .CertificateChain' > tls.crt
    aws acm export-certificate --certificate-arn ${CERTARN} --passphrase ${TLS_SERVER_PROFILE_PASSPHRASE} --cli-binary-format raw-in-base64-out | jq -r '"\(.CertificateChain)"' > ca.crt

create kubernetes secret - kubectl create secret -n my-namespace generic my-istio-cert --from-file=tls.key=tls.key --from-file=tls.crt=tls.crt --from-file=ca.crt=ca.crt

Istio configuration -

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - 'myhost.com'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      cipherSuites:
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      credentialName:  my-istio-cert
      minProtocolVersion: TLSV1_2
      mode: SIMPLE