Using vault values in health checks

[far-blue]

Nomad version

Nomad v0.11.1 (b43457070037800fcc8442c8ff095ff4005dab33)

Operating system and Environment details

Ubuntu 18.04

Issue

It doesn't seem possible to use vault values such as passwords in defining service health checks - such as Authorization header values for http health checks.

From what I can tell, Vault values are only available via consul-template which is only allowed to write files or set env vars in the target container. Otoh, the service health checks don't have visibility of the environment vars set in the target container.

Reproduction steps

I've tried various options and nothing has worked. Note that normal interpolation - such as ${NOMAD_GROUP_NAME} does work and that's not what I'm reporting here.

Job file (if appropriate)

This is how I feel it could work (but it doesn't). This example tries to setup the basic auth header for an http based health check. First we create the base64 value:

            template {
                change_mode = "noop"
                data = <<EOF
{{ with secret "kv/app/settings" }}
AuthHeader = {{ printf "%s:%s" .Data.data.username .Data.data.password | base64URLEncode }}
{{ end }}
EOF
                destination = "secrets/env"
                env = true
            }

Then we'd want to use it within a health check:

            service {
                check {
                    type = "http"
                    port = 1234
                    path = "/my/healthcheck"
                    header {
                        Authorization = ["Basic ${AuthHeader}"]
                    }
                    interval = "10s"
                    timeout = "2s"
                }
            }

Am I missing something? Should this work? Is there a different way to achieve this? I know that needing auth for a health check isn't great but some apps like RabbitMQ require auth for their API and I'd prefer not to have the password in the Job file.

[rkettelerij]

While it would be nice to use Vault secrets in HTTP or TCP health checks there is a workaround for this: use a script check.

• Use the template stanza to generate a simple bach script that performs a curl on /my/healthcheck. The template stanza has access to Vault secrets so providing the basic auth header shouldn't be a problem.

• Change your check to a script check and use the bash script you created with the template stanza as input.

[far-blue]

@rkettelerij thanks for the comment - yeah, I did end up using a script check but using wget rather than curl because curl isn't in docker image I'm using. I also decided to leave my template stanza setting env vars as before and then just used the $AuthHeader value in the script command string as I think it makes things a bit more obvious to others reading the job file.

However, I definitely think using a script check in this case is a bit of a hack - after all you could argue there's no point having http, grpc and tcp check types because all of them can be achieved with a script check :) Script checks are also not supported by the qemu driver so not a universal fix.

It would be very helpful if template-originating env values could also then be used during interpolation.