A proxy-wide health check route

[igor-alexandrov]

Cloud load balancers like AWS ALB or DigitalOcean droplet balancer does not allow specifying HOST header that is getting sent to the instance. In a multi-application setup, this means two things:

1. We should always have once application in kamal-proxy with proxy: { host: nil } setting.
2. The health status of the instance is determined from the status of a single application, which is wrong.

Based on these two facts, I suggest adding an ability to specify a proxy-wide health check route that will work for all instances.

[kczpl]

Hi! I've encountered the same problem and would be happy to contribute to solving it.

Before Kamal v2.0, I resolved this issue using Traefik labels to route traffic based on specific User-Agent headers. Since I have both a FE and BE applications hosted on the same server, I need to separate my health checks:

 traefik.http.routers.app-web-staging.rule: "Host(`api.domain.com`) || (Headers(`User-Agent`, `ELB-HealthChecker/2.0`) && PathPrefix(`/health`))"

I believe we should add an option that allows routing based on headers and paths, not just the host, as I don’t see any other way to handle managed load balancers' health checks due to the limited request information they provide.

Alternatively, we could add a way to easily expose the container port, like 0.0.0.0:3000, and target the health check there. Since we're dealing with a managed service, which is usually protected by security groups and other measures.

[igor-alexandrov]

As a workaround for now, you can deploy the smallest (around 80Kb) image https://hub.docker.com/r/lipanski/docker-static-website without the host option and then deploy all your apps as usual with the host option.

[kczpl]

Thank you! I temporarily changed the range (200-499) of my health checks since it’s pointless anyway, but I didn’t know that image exists. :))

[dhh]

Help me understand the problem here. The proxy/ssl|host settings are only relevant when you're running with a single box. Why would there be a load balancer in front then?

[igor-alexandrov]

The problem is that almost all balancers allow to define only an HTTP path and port as a healthcheck. This works well when you have a single app on the host.

Imagine yourself having a balancer. It sends a /up HTTP GET request without a HTTP_HOST header (because your web interface does not allow you to configure this) to check that your instance is up and running. kamal-proxy receives the request and tries to respond, but because you have multiple app containers configured and each of them has a host option populated (which makes sense) it cannot find a container to process the request.

Currently, kamal-proxy will respond with HTTP 404, which makes sense because it will not wind a container to route traffic to.

The idea is to add a default traffic point like /up that will respond with the host machine HTTP status when no HTTP_HOST header is defined.

[Rohland]

Help me understand the problem here. The proxy/ssl|host settings are only relevant when you're running with a single box. Why would there be a load balancer in front then?

Running into this with the upgrade to 2.0 (very excited to drop Traefik!). With Kamal 1*, we have TLS terminating on our load balancers (AWS ELB), however, we maintained TLS right through to hosts using self signed certificates with Traefik. This ensures traffic is encrypted from client => load balancer => host. In some ways you're right, it doesn't make sense to leverage Kamal Proxy's auto SSL stuff in this setup, but it would be nice to support encryption across all network hops whilst keeping the proxy set up simple.

17 might address this scenario, but still requires setting a host (as far as I can tell). Not sure it will work with something like the following. Will test this once kamal supports the custom TLS stuff that appears to now be supported in Kamal Proxy.

proxy:
  ssl: true
  ssl_certificate: /....
  ssl_private_key: /...
  hosts: *

[kczpl]

The proxy/ssl|host settings are only relevant when you're running with a single box. Why would there be a load balancer in front then?

I have applications deployed horizontally, with three web app servers behind an AWS ALB, which handles SSL termination and health checks.

The ALB health check request looks like this:

GET / HTTP/1.1
Host: 172.x.x.x
Connection: close
User-Agent: ELB-HealthChecker/2.0
Accept-Encoding: gzip, compressed

unfortunately, there is no option to add custom headers (like Host) to the managed health checks (damn you, AWS). The issue is that I want to route this managed health check to my app so that it can respond with whether the app is healthy or not. However, I don't currently see a way to do this.

[mooktakim]

This is related to the issue I submitted in the kamal repo: https://github.com/basecamp/kamal/issues/1096

I think the solution should be to add a "default" host to the proxy.

Useful also when you want an app to show when you visit the IP address.