[pdc-describe] Use health check for nginx

[carolyncole]

What maintenance needs to be done?

When pdc-describe's health checks are failing (health.json), it should not serve that application to the public.

Level of urgency

• [x] High
• [ ] Moderate
• [ ] Low

Why is this maintenance needed?

This will make it easier and safer to deploy to only some of the machines, because we won't have to adjust the nginx configuration to take things off the load balancer just in case it breaks.

Acceptance criteria

• [x] When one node is broken it does not get traffic diverted to it.
• [x] It checks /health.json

Implementation notes, if any

Nginx Documentation: https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/ Existing health check configuration: https://github.com/pulibrary/princeton_ansible/blob/main/roles/nginxplus/files/conf/http/lib-jobs-staging.conf

First Step

Add the health check URL to the staging configuration.

See https://github.com/pulibrary/princeton_ansible/pull/5223/files for an example

[jrgriffiniii]

This seems to already be in place for production and staging:

https://github.com/pulibrary/princeton_ansible/blob/main/roles/nginxplus/files/conf/http/pdc-describe_prod.conf#L81

[carolyncole]

@jrgriffiniii that is not checking health.json

[carolyncole]

This change is online and we can see the sidekiq console again utilizing capistrano, but nginx is not allowing requests to hit pdc-describe-prod2