Openshift: Ingress not working out of the box

[mhutter]

Steps to Reproduce the Problem

1. Prepare TLS cert as documented in https://hub.syn.tools/keycloak/how-tos/keycloak-tls.html
2. Deploy keycloak to OpenShift 4.8
3. Access ingress

Actual Behavior

502 Bad Gateway

The server returned an invalid or incomplete response.

Expected Behavior

The application is served

[mhutter]

So 2 things here:

1. When converting from an Ingress to a route, the nginx.ingress.kubernetes.io/backend-protocol: HTTPS annotation is not honored and the resulting Route always uses edge termination -> report as bug to Red Hat, but in the mean time we probably want to implement this in the component instead.
2. The certificate's CN must equal the service's short name, eg. keycloak-http.<namespace>.svc and be generated by OpenShift, or otherwise the cert's root must be added to the route. Service serving certificate secrets can be used for this

[mhutter]

Opened Red Hat case #03068352 for the original issue and preparing a workaround in the component.

[ccremer]

Can you share the parameters you used for an installation on OCP 4? Because APPUiO Cloud is using parameters that work for an ingress, so Ingress works but not quite "out of the box". Did you also consider https://hub.syn.tools/keycloak/how-tos/openshift-4.html?

When converting from an Ingress to a route, the nginx.ingress.kubernetes.io/backend-protocol: HTTPS annotation is not honored

Yes, that's expected, as its an annotation meant for the nginx controller. It doesn't hurt either and we didn't yet invest much into cleaning up unsupported annotations when a certain distribution is targeted. This component has default values targeting a non-Openshift cluster with nginx as ingress controller.

[mhutter]

Can you share the parameters you used for an installation on OCP 4?

Sure

```yaml keycloak_test: namespace: keycloak-test ingress: servicePort: http fqdn: test.anmeldung.example.ch admin: username: '?{vaultkv:${cluster:tenant}/${cluster:name}/keycloak-test/admin-username}' password: '?{vaultkv:${cluster:tenant}/${cluster:name}/keycloak-test/admin-password}' database: tls: enabled: false provider: external external: host: maxscale-masteronly vendor: mariadb port: 3306 database: keycloak_test username: keycloak_test extraInitContainers: theme-provider: image: image-registry.openshift-image-registry.svc:5000/tpl-build-integration/example-keycloak-theme:test imagePullPolicy: Always command: - sh args: - -c - | echo "Copying theme..." cp -Rv /themes/* /target/ volumeMounts: - name: themes mountPath: /target extraVolumes: themes: emptyDir: {} helm_values: image: tag: 14.0.0 podSecurityContext: null securityContext: null pgchecker: securityContext: null extraVolumeMounts: | - name: keycloak-tls readOnly: true mountPath: /etc/x509/https - name: themes mountPath: /opt/jboss/keycloak/themes/example subPath: example - name: themes mountPath: /opt/jboss/keycloak/themes/app1 subPath: example # ... snip ... ```

Did you also consider https://hub.syn.tools/keycloak/how-tos/openshift-4.html?

Yes, that's what we used to configure initially.

The setup where we discovered those issues have been removed again, but we'll set up a development environment that we can use for investigations soon. Once this is done I'll send a bunch of PRs

[ccremer]

I did spot an issue :see_no_evil: It's a classic YAML indentation problem. The ingress.servicePort should be beneath helm_values. This is wrong in the docs. The config should be:

  keycloak_test:
    ...
    helm_values:
      ingress:
        servicePort: http

Once you fixed this, can you please try again? Meanwhile I'll open a PR

[mhutter]

This just saved me so much time for investigation 😂 Thanks!

[ccremer]

:) You might also want to set the database.jdbcParams to empty string, if you don't want any encryption to DB (in your case maxscale)

[anothertobi]

Would adding route.openshift.io/termination: "reencrypt" [1] to keycloak:ingress:annotations work in your use-case with OpenShift? If so, we could think about adding this to the defaults.

IMHO converting nginx.ingress.kubernetes.io/backend-protocol: HTTPS should not be done when translating an Ingress to a Route, as it is an annotation specific to NGINX. Otherwise all annotations for the different Ingress implementations would have to be translated.

[1] https://docs.openshift.com/container-platform/4.8/networking/routes/route-configuration.html#nw-ingress-creating-a-route-via-an-ingress_route-configuration

[ccremer]

Is this still an issue here?

[mhutter]

Will review when improving OCP4 compatibility

[mhutter]

Adding route.openshift.io/termination: "reencrypt" to the Ingress object fixes the Route to correctly use "reencrypt" as its tlsTermination mode, but still yields an "Application is not available" when accessing it.

I suppose this is caused by the router not trusting the upstream certificate.

I'll try to use a service serving cert instead (which is trusted by default by the router)

[mhutter]

Also, I suppose disabling cluster-internal TLS would work but that kinda misses the point of having it in the first place?

[ccremer]

We've not invested time to make it work with Openshift + Ingress + reencrypt...

If you do find a working solution, would you mind sharing it?

[mhutter]

Ah I think I have it figured out

• Don't create keycloak-tls secret from vault or cert-manager
• Add the service.beta.openshift.io/serving-cert-secret-name: "keycloak-tls" annotation to the keycloak-http service
• Set route to "reencrypt"
• profit

I'm currently testing this on our dev instance and will send a PR when ready

[mhutter]

Have some PRs :P #89 #90 #91 #92

They are based off each other to avoid merge conflicts on everyone so make sure to rebase them before merging