LB SSL certificates cannot be updated without deleting the service

[prydie]

Currently the TLS secrets for load balancers are not checked and changes reflected in OCI.

[jhorwit2]

I was chatting with some team members, the only way to do this currently is to create a new certificate and update the listener(s). It's still an open question if this results in any downtime though.

[Sugi275]

Hi, I found a way to update SSL certificate to no downtime. But, I don't test run, because I don't have Kubernetes Cluster with OCI. Would you please confirm it?

Probably need to enhance this function. https://github.com/oracle/oci-cloud-controller-manager/blob/master/pkg/cloudprovider/providers/oci/load_balancer_util.go#L411

Now Code

func getListenerName(protocol string, port int, sslConfig *loadbalancer.SslConfigurationDetails) string {
    if sslConfig != nil {
        return fmt.Sprintf("%s-%d-%s", protocol, port, *sslConfig.CertificateName)
    }
    return fmt.Sprintf("%s-%d", protocol, port)
}

Enhanced code

delete CertificateName.

func getListenerName(protocol string, port int, sslConfig *loadbalancer.SslConfigurationDetails) string {
    return fmt.Sprintf("%s-%d", protocol, port)
}

steps to update SSL certificate

First, create certificate to OKE

kubectl create secret tls ssl-certificate-secret --key privkey.pem --cert cert.pem

apply nginx Deployment and ServiceLoadBalancer to OKE.

cat <<'EOF' > ~/workdir/nginx-https.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-service
  annotations:
    service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/oci-load-balancer-tls-secret: ssl-certificate-secret
spec:
  selector:
    app: nginx
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
EOF
kubectl apply -f ~/workdir/nginx-https.yaml

show OCI LB. listener(TCP-443-ssl-certificate-secret) is created in OCI LoadBalancer.

> oci lb load-balancer list
...snip...
      "listeners": {
        "TCP-443-ssl-certificate-secret": {     <======== lister name fotmat '("%s-%d-%s", protocol, port, *sslConfig.CertificateName)'
          "connection-configuration": {
            "idle-timeout": 300
          },
          "default-backend-set-name": "TCP-443",
          "hostname-names": null,
          "name": "TCP-443-ssl-certificate-secret",
          "path-route-set-name": null,
          "port": 443,
          "protocol": "TCP",
          "rule-set-names": [],
          "ssl-configuration": {
            "certificate-name": "ssl-certificate-secret",
            "verify-depth": 0,
            "verify-peer-certificate": false
          }
        },
...snip...

create new certificate to OKE, to update SSL Certificate.

kubectl create secret tls ssl-certificate-secret-next --key privkey.pem --cert cert.pem

apply new Service to OKE.

update annotations service.beta.kubernetes.io/oci-load-balancer-tls-secret: ssl-certificate-secret-next

cat <<'EOF' > ~/workdir/nginx-https.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
kind: Service
apiVersion: v1
metadata:
  name: nginx-service
  annotations:
    service.beta.kubernetes.io/oci-load-balancer-ssl-ports: "443"
    service.beta.kubernetes.io/oci-load-balancer-tls-secret: ssl-certificate-secret-next
spec:
  selector:
    app: nginx
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: 80
  - name: https
    port: 443
    targetPort: 80
EOF
kubectl apply -f ~/workdir/nginx-https.yaml

Downtime will occur at this time. Because old listener is deleted and new listener is created. I guess that it will solve by deleting CertificateName from Listener name.

> oci lb load-balancer list
...snip...
      "listeners": {
        "TCP-443-ssl-certificate-secret-next": {     <======== lister name fotmat '("%s-%d-%s", protocol, port, *sslConfig.CertificateName)'
          "connection-configuration": {
            "idle-timeout": 300
          },
          "default-backend-set-name": "TCP-443",
          "hostname-names": null,
          "name": "TCP-443-ssl-certificate-secret-next",
          "path-route-set-name": null,
          "port": 443,
          "protocol": "TCP",
          "rule-set-names": [],
          "ssl-configuration": {
            "certificate-name": "ssl-certificate-secret-next",
            "verify-depth": 0,
            "verify-peer-certificate": false
          }
        },
...snip...

[jhorwit2]

@Sugi275 Just an FYI that unless something has changed recently with LBaaS it's my understanding that any update to the certificate will cause a momentary blip of downtime regardless of the update pattern.

[Sugi275]

@jhorwit2 Thank you reply!

In my comment pattern, there is downtime of 10 seconds or more. my comment pattern : https://github.com/oracle/oci-cloud-controller-manager/issues/14#issuecomment-462461751 (Recreate Listener pattern)

In the pattern that does not recreate Listener, it will be a momentary downtime or no downtime.

[razitz]

@prydie @jhorwit2

We're facing similar issue as well, after changing "service.beta.kubernetes.io/oci-load-balancer-tls-secret" for the second time, we keep getting:

Warning CreatingLoadBalancerFailed 7s (x12 over 23m) service-controller Error creating load balancer (will retry): failed to ensure load balancer for service <namespace>/<resource name>: updating listener: Service error:InvalidParameter. Default Listener on port '443' refer to VIP 'private-vip' twice. http status code: 400

[shyamradhakrishnan]

@razitz We will have to treat your scenario as a separate issue i.e. if you update the tls secret name, it should work and should not fail the second time. Could you please open a separate issue for this? cc @arindam-bandyopadhyay @AkarshES

[algora]

Opened https://github.com/oracle/oci-cloud-controller-manager/issues/344 for the issue described in https://github.com/oracle/oci-cloud-controller-manager/issues/14#issuecomment-762111511