Modification to instalation instructions required for installing on Auto Pilot GKE

[drrk]

On GKE in Auto Pilot mode a change to the installation instructions is required to install correctly. On this installation command, you need to append --set global.leaderElection.namespace=harmony This is to ensure that cert-manager is correctly setup due to a security limitation imposed by Google. Seehttps://github.com/cert-manager/cert-manager/issues/3717#issuecomment-931567578 for further information.

[bradenmacdonald]

Thanks for the report @drrk! I'll come up with a fix and get your help testing it.

[drrk]

You are welcome - there might be a bit more to it - whilst this meant the helm chart installed correctly, when I then deployed an instance with tutor it didn't get an SSL certificate correctly, it was still on the fake ingress certificate. I don't know if this was user error, or if there is still some missing steps to get cert-manager functional. I am going to test again tomorrow (I am in the UK)

[drrk]

I've tested again, using a different method, which I thought should work from looking at the cert-manager documentation https://cert-manager.io/docs/installation/compatibility/#gke-autopilot and https://cert-manager.io/docs/installation/helm/#installing-cert-manager-as-subchart

I set the relevant section of values.yaml as such:

cert-manager:
  # Set your email address here so auto-generated HTTPS certs will work:
  email: "kimball@appsembler.com"
  namespace: harmony
  global.leaderElection.namespace: harmony

However this didn't work at all, the helm chart failed to instal, as it would without trying anything:

Error: INSTALLATION FAILED: failed post-install: 1 error occurred:
        * timed out waiting for the condition

I think another option is to try installing cert-manager separately first, and then installing harmony without cert-manager. I will try that when I get a chance.

[drrk]

Okay, I have it working by installing cert-manager first, then installing harmony, then manually adding the Issuer config. However, I think it might also work with my initial step of just adding to the helm command line. The part I missed before is you need an additional DNS entry of app.<lmsfqdn> for cert-manager to get the certificate correctly.

I can do any more tests you need, but only for the next 10 day after which I will change job, but @amirtds should be able to followup.

[drrk]

My process was as follows

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.11.1 \
  --set installCRDs=true \
  --set global.leaderElection.namespace=cert-manager

After this completes successfully, harmony should be installed, with cert-manager.enabled: false set in the values.yaml as documented in GitHub - openedx/openedx-k8s-harmony: A Prototype Helm Chart for deploying multiple Open edX instances (via Tutor) onto a cluster.

Once this is complete, the cert-manager issuer config is needed based on this template: openedx-k8s-harmony/issuer.yaml at main · openedx/openedx-k8s-harmony

This needs to be installed after the harmony chart, as it depends on nginx-ingress

An example file (named issuer.yaml) is:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: harmony-letsencrypt-global
spec:
  acme:
    email: user@example.com
    preferredChain: ""
    privateKeySecretRef:
      name: harmony-letsencrypt-global
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx

which is installed with kubectl apply -f issuer.yaml

[bradenmacdonald]

Thanks for the updates @drrk! I'm sorry for the delay here; I'll try to incorporate this next week.

[drrk]

No worries at all. As I said before, Friday is my last day at Appsembler, and thus working with Open edX for now, but hopefully this will help others. I know others at Appsembler will be following up too.

[felipemontoya]

@drrk nothing has moved in this issue for a while and nobody from Appsembler has reached out. Are you ok if we close this issue for the time being?

[drrk]

Go ahead and close it. Sorry no one else has reached out.

Kimball

[felipemontoya]

Thanks. Closing now