[GCP] Make sure the GCE ingress uses the defined readiness probe definition

[guidoiaquinti]

Bug description

While working on the e2e test for GCP, I found the GCE Ingress (GLB) not picking up the health check information from the readiness probe definition. Instead, it is using the default / making the health check to fail (as the app returns an HTTP 302).

This StackOverflow thread might have more info.

Expected behavior

When we deploy PostHog to GCP, the ingress load balancer uses the HTTP /_health path as readiness probe.

Actual behavior

When we deploy PostHog to GCP, the ingress load balancer uses the HTTP / path as readiness probe.

How to reproduce

1. Install PostHog on GCP

[guidoiaquinti]

Note for myself: once the issue is fixed, we should re-enable the ingestion test we disabled:

https://github.com/PostHog/charts-clickhouse/blob/e0a2f06d61fd67524ef538c132310244719024f9/.github/workflows/test-google-cloud-platform-install.yaml#L158-L167

[georgekaz]

Is there any workaround for this other than manually adding /_health to the health check in the GCP console? Will you add backend config https://cloud.google.com/kubernetes-engine/docs/concepts/ingress#direct_hc and add the cloud.google.com/backend-config annotation as per https://cloud.google.com/kubernetes-engine/docs/how-to/ingress-features#create_service ?

Thanks

[guidoiaquinti]

👋 Hey @georgekaz , unfortunately I wasn't able to push a workaround/fix for this in https://github.com/PostHog/charts-clickhouse/pull/209. The task is currently in our backlog and hasn't been prioritised.

Let us know if you find a way to make it working and I'll be happy to push the change. I'm unfortunately pretty new to GKE.

Thank you!

[georgekaz]

I couldn't find a way to make this work without too much bother so I decided to use nginx ingress instead.