On OpenShift, the Console and Keycloak currently rely on the automatic injection of service-ca.crt into the pod in order to trust the the Kubernetes Auth Server and the Keycloak API server (respectively). This feature is deprecated and is due to go-way.
The alternative mechanism for getting the service-ca.crt is described here:
In addition, EnMasse currently makes use of service.alpha.openshift.io/serving-cert-secret-name. This has been replaced with service.beta.openshift.io/serving-cert-secret-name. The semantics appears to be the same.
When running on OpenShift 4, the newer features should be used. Support for OpenShift 3.11 is required, so behaviour should unchanged when running there.
Design
EnMasse Operator
On OpenShift4 the controller loop will reconcile a service-ca configmap with the service.beta.openshift.io/inject-cabundle annotation.
The operator deployment (yaml bundle/CSV) will declare this confgmap as an optional: true volume.
On first use the configmap won't exist so the controller will create it The controller will then check for the appearance of the projected service-ca.crt on the pod's filesystem. If the file does not exist yet, the reconcile loop will be rescheduled with a requeueafter to cause the controller to await the projected file. This will have the affect of delaying the operator's useful work util after the service-ca.crt is available to it. This relies on https://github.com/kubernetes/website/issues/26611.
To be safe, we might chose to fail the operator process entirely if the projected file does not appear with a certain number of iterations.
The REST API calls used to configure Keycloak that currently rely on /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt need to change to use the configmap's mount point.
Console
On OpenShift, currently the /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt is passed as part of oauth-proxy-openshift.cfg as an upstream configuration value.
On OpenShift4 the console reconciliation needs to change so that the console deployment mount the service-ca configmap. This doesn't need to be optional as by this point the service-ca configmap can be assumed to exist.
The deployment's arguments to oauth-proxy need to pass the --upstream argument with either the configmap's mount point or fall back to /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt on OpenShift 3.11.
Keycloak
The AddressSpaceController uses Keycloak REST to create/delete realms. The Keycloak server uses a serving-cert-secret-name annotated service, so needs the service-ca.crt in order to conect.
The solution here mirrors the Console. The address-space-controller deployment needs the service-ca configmap mounted volume on the OpenShift 4 path. The KubeKeycloakFactory need to use either the configmap's mount poin or fall back to /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt on OpenShift 3.11.
Serving Cert Changes
In the EnMasse Operator, on OpenShift 4 must reconcile the console and authservices services to use the service.beta.openshift.io form of the annotation name. If the controller detects the service.alpha.openshift.io form it should remove it.
The AddressSpace AuthController OpenShiftProvider issues certificates using service.alpha.openshift.io. It needs to change too on OpenShift4.
Service Broker
The Service Broker is no longer supported on OpenShift 4.6, so it will continue to use service.alpha.openshift.io to continue to support OpenShift 3.11.
If applicable, links to design docs.
Tasklist
Enumerate the sub-tasks required to complete the task:
vbusch
commented
3 years ago
Description
On OpenShift, the Console and Keycloak currently rely on the automatic injection of
service-ca.crtinto the pod in order to trust the the Kubernetes Auth Server and the Keycloak API server (respectively). This feature is deprecated and is due to go-way.The alternative mechanism for getting the service-ca.crt is described here:
https://docs.openshift.com/container-platform/4.6/security/certificates/service-serving-certificate.html#add-service-certificate-configmap_service-serving-certificate
In addition, EnMasse currently makes use of
service.alpha.openshift.io/serving-cert-secret-name. This has been replaced withservice.beta.openshift.io/serving-cert-secret-name. The semantics appears to be the same.https://docs.openshift.com/container-platform/4.6/security/certificates/service-serving-certificate.html#add-service-certificate_service-serving-certificate
When running on OpenShift 4, the newer features should be used. Support for OpenShift 3.11 is required, so behaviour should unchanged when running there.
Design
EnMasse Operator
On OpenShift4 the controller loop will reconcile a service-ca
configmapwith the service.beta.openshift.io/inject-cabundle annotation. The operator deployment (yaml bundle/CSV) will declare thisconfgmapas anoptional: truevolume.On first use the configmap won't exist so the controller will create it The controller will then check for the appearance of the projected
service-ca.crton the pod's filesystem. If the file does not exist yet, the reconcile loop will be rescheduled with a requeueafter to cause the controller to await the projected file. This will have the affect of delaying the operator's useful work util after the service-ca.crt is available to it. This relies on https://github.com/kubernetes/website/issues/26611.To be safe, we might chose to fail the operator process entirely if the projected file does not appear with a certain number of iterations.
The REST API calls used to configure Keycloak that currently rely on
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crtneed to change to use the configmap's mount point.Console
On OpenShift, currently the
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crtis passed as part ofoauth-proxy-openshift.cfgas anupstreamconfiguration value.On OpenShift4 the console reconciliation needs to change so that the console deployment mount the service-ca
configmap. This doesn't need to be optional as by this point the service-caconfigmapcan be assumed to exist.The deployment's arguments to oauth-proxy need to pass the --upstream argument with either the configmap's mount point or fall back to
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crton OpenShift 3.11.Keycloak
The AddressSpaceController uses Keycloak REST to create/delete realms. The Keycloak server uses a serving-cert-secret-name annotated service, so needs the service-ca.crt in order to conect.
The solution here mirrors the Console. The address-space-controller deployment needs the service-ca
configmapmounted volume on the OpenShift 4 path. The KubeKeycloakFactory need to use either the configmap's mount poin or fall back to/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crton OpenShift 3.11.Serving Cert Changes
In the EnMasse Operator, on OpenShift 4 must reconcile the console and authservices services to use the
service.beta.openshift.ioform of the annotation name. If the controller detects theservice.alpha.openshift.ioform it should remove it.The AddressSpace AuthController OpenShiftProvider issues certificates using service.alpha.openshift.io. It needs to change too on OpenShift4.
Service Broker
The Service Broker is no longer supported on OpenShift 4.6, so it will continue to use service.alpha.openshift.io to continue to support OpenShift 3.11.
If applicable, links to design docs.
Tasklist
Enumerate the sub-tasks required to complete the task: