search

opentargets / issues

Issue tracker for Open Targets Platform and Open Targets Genetics Portal
https://platform.opentargets.org https://genetics.opentargets.org
Apache License 2.0
12 stars 2 forks source link

Open Search readiness sidecar #3245

Open mbdebian opened 6 months ago

mbdebian commented 6 months ago

When deploying Open Targets Platform Open Search data backend, our current infrastructure waits for a fix amount of time before probing new instances on readiness, and health checks are based on Open Search being up and running, but the service has two different stages:

  1. When the engine is up and running, which happens very quickly as soon as the docker image is run in the VM instance
  2. When the actual Open Search cluster is ready, this parameter presents three different values: red, yellow and green. In order for the cluster to be ready for serving data requests, the reported status must be green (yellow would also serve data requests, but in a degraded state, i.e. less performant)

We need a sidecar docker image that can probe the main docker image running within the same instance and docker network, for cluster readiness, and then use this mechanism as a health and readiness check in the infrastructure.

jdhayhurst commented 5 months ago

I don't know if this meets the needs exactly, but there is an api for checking the health status and waiting for it to be green: _cluster/health?wait_for_status=green - what do you think @mbdebian?

javfg commented 4 months ago

I don't know if this meets the needs exactly, but there is an api for checking the health status and waiting for it to be green: _cluster/health?wait_for_status=green - what do you think @mbdebian?

This is a good find, we discussed about it but we found a problem: there is no way to parse such a response from a probe, which expects just a HTTP 200 if things are OK.

We decided to create a sidecar that periodically checks the status and publishes a simpler response:

https://github.com/opentargets/opensearch-nanny