This change adds the option to select a HTTP healthcheck for a services target group. The reason for this change is to avoid downtime during deploys, as at present the TCP healthcheck will only wait for nginx to be able to open a connection although the underlying scala service may not yet have started.
[!NOTE]
The intention is to roll this our for the catalogue-api works service to start with, and then add the necessary endpoints to the items and concepts service, finally removing the option for a TCP health-check. In addition further work is required to properly report the health of the service as being able to respond from the app container isn't definitive proof the app is "healthy".
How can we test?
[x] Manually check this healthcheck behaves as expected in the stage environment and tasks register as healthy
[x] Deploy this change to stage to ensure the terraform applies the change as expected and tasks register as healthy
[x] Perform a stage deployment while sending requests to understand if we have eliminated the deployment errors.
How can we measure success?
No downtime during deployments resulting in a better experience for visitors to the site, and fewer errors that we cannot effectively respond to in the alerts channel.
Have we considered potential risks?
Changing the health-checks changes the failure modes for the API, we should test thoroughly in stage before deploying to prod, consider and document the impact of extending the health check to fail in other situations (e.g. elasticsearch is unavailable).
What does this change?
This change adds the option to select a HTTP healthcheck for a services target group. The reason for this change is to avoid downtime during deploys, as at present the TCP healthcheck will only wait for nginx to be able to open a connection although the underlying scala service may not yet have started.
By default the HTTP healthcheck uses the
/management/healthcheckendpoint (already present in the works service), and will register as healthy if that endpoint returns a 200 (see https://api.wellcomecollection.org/catalogue/v2/management/healthcheck).How can we test?
How can we measure success?
No downtime during deployments resulting in a better experience for visitors to the site, and fewer errors that we cannot effectively respond to in the alerts channel.
Have we considered potential risks?
Changing the health-checks changes the failure modes for the API, we should test thoroughly in stage before deploying to prod, consider and document the impact of extending the health check to fail in other situations (e.g. elasticsearch is unavailable).