[allsearch-api] Use health.json for health checks

[kayiwa]

What maintenance needs to be done?

When allsearch-api's health checks are failing (health.json), it should not serve that application to the public.

Level of urgency

• [ ] High
• [x] Moderate
• [ ] Low

Why is this maintenance needed?

This will make it easier and safer to deploy to only some of the machines, because we won't have to adjust the nginx configuration to take things off the load balancer just in case it breaks.

Acceptance criteria

• [x] When one node is broken it does not get traffic diverted to it.
• [x] It checks /health.json?providers[]=database&providers[]=solr

Implementation notes, if any

Nginx Documentation: https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/ Existing health check configuration: https://github.com/pulibrary/princeton_ansible/blob/main/roles/nginxplus/files/conf/http/imagecat_staging.conf#L35

First Step

Add the health check URL to the staging configuration.

[rladdusaw]

We decided to use /health.json to future proof further additions to the health check. It returns 200 if everything is running and an error if any of the services are down.