search

fabiolb / fabio

Consul Load-Balancing made simple
https://fabiolb.net
MIT License
7.25k stars 620 forks source link

Unable to load correct certificates if 1 invalid one is in consul k/v #941

Open finwo opened 4 months ago

finwo commented 4 months ago

Summary

When there's a single invalid certificate in the consul certificate store in use by fabio, all the other certificates in the store will not be refreshed.

Upon a fresh start of fabio (like a server reboot), this causes no single ssl endpoint to function anymore.

Steps to reproduce

We're running fabio using the fabiolb/fabio:latest image from dockerhub on a nomad cluster as system job (running on every worker node).

First off, here's the used config loaded into /etc/fabio/fabio.properties within the container from a nomad template:

proxy.addr = :{{env "NOMAD_PORT_http"}};proto=http,:{{env "NOMAD_PORT_https"}}:proto=https;cs=cssl
ui.addr = :{{env "NOMAD_PORT_ui"}}
registry.consul.register.enabled = false
{{with nomadVar "nomad/jobs/fabio"}}
registry.consul.addr = {{.consul_addr}}
proxy.cs = cs=cssl;type=consul;cert=http://{{.consul_addr}}/v1/kv/certs
{{end}}

The certificates are loaded into /certs within consul as bundles, so /certs/example.com.pem contains the full chain including the private key.

To replicate the issue, insert a single certificate where the private key does not match the certificate. This will result in fabio not loading any certificates.

Expected result

Fabio to load all valid certificates, and ignore/skips the invalid certificates, allowing domains where valid certificates are available for to continue to function.

Actual result

Fabio refuses to load or update any certificate, causing all services it's loadbalancing to fail upon ssl initialization because it fails to find a valid certificate for the servername indicated by SNI.