Https health check registration with Consul is not working

[mikehwang]

I tried running my container with the -e flag to pass the environment variables SERVICE_443_CHECK_HTTPS and SERVICE_CHECK_HTTPS. The container registers with Consul but there's no HTTPS health checks. I know this because I don't see the requests come in and I also verified by checking Consul via /v1/health/service endpoint.

Plain old HTTP health checks work.

• What version of Docker are you running?

  $ docker --version
  Docker version 1.12.0, build 8eab29e

• What version of registrator are you running? I just pulled the latest tag which is on v7.

• What is the exact command you are running registrator with?

  docker run -d \
  --name=registrator \
  --net=host \
  --volume=/var/run/docker.sock:/tmp/docker.sock \
  gliderlabs/registrator:latest \
    -ip=<external ip of my local machine> \
    consul://<consul server>:8500

The Consul server version:

$ docker exec a80d2e4f833fc3095fcf913ca1880035758962a222619235cbbb03f4b275d379 consul --version
Consul v0.6.4
Consul Protocol: 3 (Understands back to: 1)

[mevric]

@mikehwang any luck with finding a workaround this issue?

[minhkct]

Had the same issue. http works but not for https. I used script check for workaround but it's not a good solution. -e "SERVICE_8443_CHECK_SCRIPT=curl -s https://$HOSTNAME:$PORT/health" \

[minhkct]

Found the reason in my case. I used https check but did not setup TLS correctly. For ignoring TLS, need to do some small change in the source code. The solution #447 was suggested sometime ago but it was not merged to the master branch yet.

I made change similarly plus a small fix

Add the env variable SERVICE_443_CHECK_TLS_SKIP_VERIFY=true

---- consul.go ----

import "strconv"
...

func (r *ConsulAdapter) buildCheck(service *bridge.Service) *consulapi.AgentServiceCheck {
...
if  value:= service.Attrs["check_tls_skip_verify"]; value != "" {
tls_skip_verify, _ := strconv.ParseBool(value)
check.TLSSkipVerify = tls_skip_verify

...
}
}

[chrisgilbert]

I think this issue is possibly down to a change in alpine 3.5 where the ca-certificates package is no longer installed with go.

I had to fix the same thing on our fork: https://github.com/hudl/registrator/pull/11

I would suggest that unless you are using self-signed, or other invalid certs, you shouldn't really need to disable verification.

Update: See PR for the fix to the Dockerfile below.

[cwalsh]

I would suggest that unless you are using self-signed, or other invalid certs, you shouldn't really need to disable verification.

I understand the sentiment, but here's my use case:

I start containers with, for example, docker-compose. They have environment variables set like SERVICE_443_CHECK_HTTPS=/health. registrator picks this up, and registers a health check in consul against https://172.17.0.16:443/health. I'm using a wildcard certificate on *.service.consul, which is used to secure traffic between the container and a load balancer. The health check then fails because Get https://172.17.0.16:443/health: x509: cannot validate certificate for 172.17.0.16 because it doesn't contain any IP SANs AFAIK IP SANs don't support wildcard IP addresses, but the IP address that could be given to the container is effectively random. Therefore I need to skip verification of the certificate for the health check.

Are there any other options that work in this case?

[chrisgilbert]

@cwalsh I think you are right to do that. Unless the cert is created with that IP it's always going to be invalid.

I was referencing another bug which had meant root certificates were no longer being installed - which meant all certificates were invalid.

I just didn't want people to be trying to work around a bug I'd identified and provided a PR for. But in your use case, I agree that the code change is necessary, and this is a different problem to the one I mentioned. Sorry for the confusion.

[jonfreedman]

I think the reason this doesn't work is pretty simple, if you look at https://github.com/gliderlabs/registrator/archive/v7.zip it does not appear the change has been released (https://github.com/gliderlabs/registrator/pull/388 was merged in July '16) but v7 was released in April '16.

We appear to be in the unfortunate situation where the documentation does not match what's actually available.

Try running gliderlabs/registrator:master instead of gliderlabs/registrator:latest

[cngo-github]

When will the next release be and the dockerhub image updated? I really need the HTTPS health checks.