How do you enable SSL support for hosted applications?

[ormico]

I have a hosted website in a k8 service exposing port 80? How do I add a certificate and enable SSL? Can I do this with a service or do I need to create an Ingress?

[ormico]

tried tackling this by creating an Ingress using the following:

apiVersion: v1
data:
  tls.crt: <certificate request file as base64>
  tls.key: <certificate p7b file as base64>
kind: Secret
metadata:
  name: webapplication2ssl
  namespace: default
type: Opaque
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: webapplication2-ingress
spec:
  tls:
  - secretName: webapplication2ssl
  backend:
    serviceName: webapplication2-svc
    servicePort: 80

this succeeded but when I run this

PS> kubectl get ing
NAME                      HOSTS     ADDRESS   PORTS     AGE
webapplication2-ingress   *                   80, 443   1h

I can see the ingress got created but it is showing no address.

I'm new at this so I might be doing it wrong, but I don't seem to be getting anywhere.

[ormico]

Update on my progress. I've tried a couple of different things. First, I tried installing nginx-ingress manually using the scripts on github, but that didn't work, though I could see what it was trying to do and it felt like progress. Then I watched a channel 9 video where someone mentioned using helm to install nginx-ingress. So I found https://docs.microsoft.com/en-us/azure/aks/kubernetes-helm . I followed those steps, which were much easier, and tried re-creating my deployment, service, and ingress but it still isn't working.

I'm getting 2 different errors. First error is an SSL error from the browser SEC_ERROR_UNKNOWN_ISSUER. Instead, of using the certificate that I passed in, it is using a certificate issued by Kubernetes Ingress Controller Fake Certificate which I assume is some kindof self signed certificate.

It probable that I've somehow misconfigured the tls secret or the Ingress but I don't know how.

Second error, is that instead of showing me the website from my container/deployment, it is just giving me the text "default backend - 404". Same as other error, I might have something misconfigured but I don't know what and don't know how to figure it out.

[ormico]

also, if I run kubectl get ing it lists 10.240.0.4 as the address, which I think is an internal k8 address. I was expecting the public IP address. Maybe I'm wrong to expect that, but it confused me. I ended up running the k8 dashboard to fidn the public IP.

[slack]

@ormico you will see the internal service address on the ingress resource unless you enable controller.publishService.enabled=true for the controller.

Default backend sounds like a routing issue, double check that the hostname is correct in the ingress resource.

Ingress Controller Fake Certificate is also usually a routing issue which pops up if hostnames can't be matched or the secrets don't contain the appropriate CN or subjectAltName.

https://github.com/kubernetes/ingress-nginx/blob/master/docs/troubleshooting.md

[ormico]

Thanks for the feedback, I just recently got this working and wanted to follow up here for anyone who comes after.

I had a variety of issues that I managed to work through. I'll list a few in case it is helpful to anyone else.

1. Certificate issues - I had a lot of issues with the certificates but mostly figuring out what type of certificate file was expected, converting the certificate that I had to that format, having it work in the browser but fail when when pod tried to communicate with another over SSL. This last one really threw me until I realized that while my browser was fine the inter-container web service call (using the public DNS) was failing because I hadn't included the intermediate certificates in the cert file when i added it to the Secret. I had to copy and past the intermediate file contents at the bottom of my PEM certificate file before I Base64ed it and pasted the Base64 text into the Secret file.

I also had some trouble with the YAML as far as figuring out the the proper way to list multiple domains. Almost all of the examples I could find dealt with a single domain and the indented nature of YAML kept biting me.

2. Proper Ingress declaration. I was only able to get this to work by declaring my ingress as a nginx.

Originally, I couldn't get it to work w/o declaring it as an nginx ingress but I may go back and try again now that I have a lot of the kinks worked out.

3. I'm currently having a problem where the Headers are too large for the nginx buffer and its causing nginx to return an error. Took me a while to realize it was an nginx problem and not a problem with my application or k8 configuration. I'm currently working on figuring out how to pass the param in the configmap to tell nginx to increase it's header buffer sizing, but I'm hoping to have that resolved soon.

Here is an example of what worked for me:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: my-ingress
spec:
  rules:
  - host: www-test.example.com
    http:
      paths:
      - backend:
          serviceName: wwwtest-svc
          servicePort: 80
        path: /
  - host: login-test.example.com
    http:
      paths:
      - backend:
          serviceName: wwwlogintest-svc
          servicePort: 80
        path: /
  - host: example.com
    http:
      paths:
      - backend:
          serviceName: www-svc
          servicePort: 80
        path: /
  - host: www.example.com
    http:
      paths:
      - backend:
          serviceName: www-svc
          servicePort: 80
        path: /
  - host: login.example.com
    http:
      paths:
      - backend:
          serviceName: wwwlogin-svc
          servicePort: 80
        path: /
  tls:
  - hosts:
    - example.com
    - www.example.com
    - login.example.com
    - www-test.example.com
    - login-test.example.com
    secretName: my-tls

[slack]

Great, thanks for the update!

Looks like you can fiddle the header buffer size by setting client-header-buffer-size in the configmap: https://github.com/kubernetes/ingress-nginx/pull/150/files

[ormico]

yeah, I found that but I'm not sure how to apply the configMap to the nginx ingress? I found a bunch of stuff that says use the configMap for this and found the setting but not how to combine the two concepts into one example and connect them together.

[slack]

Depends on how you installed nginx-ingress, the controller takes a --configmap= argument that includes the path to the configuration e.g. default/my-nginx-config.

If you installed the nginx-ingress via helm, the upstream templates interpolate the controller.config args into the configuration https://github.com/kubernetes/charts/blob/master/stable/nginx-ingress/templates/controller-configmap.yaml#L17

helm upgrade ingress stable/nginx-ingress \
--set "controller.config.client-header-buffer-size=8k,controller.config.large-client-header-buffers=16k"

Or whatever your desired values may be.

[ormico]

yeah, I did it with helm. First time I tried it manually but helm was much easier.

I think I have a parameter wrong in your example. Getting an error on the helm upgrade. Tried it with and without the first 'ingress'.

PS> helm upgrade ingress stable/nginx-ingress --set "controller.config.client-header-buffer-size=8k,controller.config.large-client-header-buffers=16k"
Error: UPGRADE FAILED: "ingress" has no deployed releases
[12/20/2017 13:16:49] D:\projects\LOS Talker\hgsvr\k8
PS> helm upgrade stable/nginx-ingress --set "controller.config.client-header-buffer-size=8k,controller.config.large-client-header-buffers=16k"
Error: This command needs 2 arguments: release name, chart path

[ormico]

this is my script to install nginx with helm, but I don't know what the 1st parameter is supposed to be in helm upgrade:

helm init
helm repo update
helm install stable/nginx-ingress
# need a pause here while install finishes
kubectl create -f www-secret.yaml
kubectl create -f wwwing-nginx.yaml

[slack]

Ah, you will need to use the release name that was generated as a result of your helm install, you should be able to figure it out doing helm ls and look for the nginx-ingress chart in the right-hand column.

Then helm upgrade <releasename> stable/nginx-ingress ...

[ormico]

ok, I might have screwed something up. I did the following but now my ingress is erroring.

PS> helm list
NAME            REVISION        UPDATED                         STATUS          CHART                   NAMESPACE
mean-ibex       1               Fri Dec 15 13:44:04 2017        DEPLOYED        nginx-ingress-0.8.21    default
[12/20/2017 13:42:21] D:\projects\LOS Talker\hgsvr\k8
PS> helm upgrade mean-ibex stable/nginx-ingress --set "controller.config.client-header-buffer-size=8k,controller.config.large-client-header-buffers=16k"
Release "mean-ibex" has been upgraded. Happy Helming!
...

[ormico]

ah, here is the error. I'm double checking the params now:

[emerg] 52#52: invalid number of arguments in "large_client_header_buffers" directive in /tmp/nginx-cfg394500534:41
nginx: [emerg] invalid number of arguments in "large_client_header_buffers" directive in /tmp/nginx-cfg394500534:41
nginx: configuration file /tmp/nginx-cfg394500534 test failed

[ormico]

ok, changed the command to the following (I'm guessing on how big the buffers need to be atm). this got the ingress working again but I'm still gettting 502 Bad Gateway. I'll drill into the ingress logs again and/or try increasing the buffers more.

helm upgrade mean-ibex stable/nginx-ingress --set "controller.config.client-header-buffer-size=16k,controller.config.large-client-header-buffers=4 32k"

[ormico]

Ok! I think I got this one. The error I was seeing in the nginx pod is: [error] 24#24: *10 upstream sent too big header while reading response header from upstream, client

I found this article: https://stackoverflow.com/questions/40289225/kubernetes-nginx-ingress-controller-returns-502-but-only-for-ajax-xmlhttprequest

So I changed the helm upgrade to: helm upgrade mean-ibex stable/nginx-ingress --set "controller.config.proxy-buffers=8 16k,controller.config.proxy-buffer-size=16k" (notice the change between _ to -) and now I'm getting past that error. (I'm getting a new error, but 1 down N to go!!!)

EDIT: I noticed that the helm line is wrapping so I wanted to make sure this part is clear. The proxy-buffers value is controller.config.proxy-buffers=8 16k Notice the space between 8 and 16k.

Thanks for the help!!!

[slack]

Awesome! 🍾

[ormico]

Is this the same way I would upgrade nginx to the latest version when new versions come out? using helm upgrade?

[slack]

Yep. helm upgrade will upgrade the chart to the latest version: https://github.com/kubernetes/helm/blob/master/docs/using_helm.md#helm-upgrade-and-helm-rollback-upgrading-a-release-and-recovering-on-failure