haproxy proxy protocol

[darix]

What would you like to be added

Add support for haproxy's proxy protocol:

https://www.haproxy.org/download/2.4/doc/proxy-protocol.txt

Why this is needed

The documentation for proxying is talking about 2 features you would like to see

1. the client should connect directly to the step-ca daemon because you use the old cert for authentication
2. you would like to see the real IP of the client

right now you need layer 4 load balancing for this. with the proxy protocol implemented, you can use a layer 7 load balancer like haproxy. the protocol is quite simple and there are already a lot of implementations for it

https://www.haproxy.com/blog/haproxy/proxy-protocol/

[darix]

Already existing implementations

• https://github.com/armon/go-proxyproto
• https://github.com/pires/go-proxyproto

[dopey]

Hey @darix we discussed this a bit last week and wanted to get a bit more info.

Main point of curiosity / contention: Why should we support layer 7 load-balancing? Most popular proxies support layer 4 proxying - I haven't checked but I'm guessing haproxy also supports layer 4 proxying based on SNI header. So, if proxying is already supported, why should we integrate further?

[darix]

yes it can do plain tcp proxying. the only reason why you want to implement their proxy protocol, is that your daemon can log the real client address instead of the proxy address.

so instead of having to do http proxying to get x-forwarded-for headers, you can use tcp proxying. see the real client certificate and the real client IP.

[maraino]

One thing to take into account is that a layer 7 proxy even with haproxy won't support the current renew/rekey endpoints. According to the spec, there's some information about the client presenting a certificate, but as I see it it won't be enough to renew or rekey a cert.

[dopey]

Great point @maraino.

We're open to this but we're interested in understanding use cases (as well as trying to gauge benefit to the community). If folks want to chime in with a +1 or tell us about a use case that would be helpful.

[darix]

proxy protocol really just means this:

PROXY TCP4 192.168.0.1 192.168.0.11 56324 443\r\n
<original content of the tcp stream>

Example is v1 protocol, v2 is binary. pires/go-proxyproto implements both.

Maybe this configuration file will help with the understanding:

global
  log stdout format short daemon
  maxconn 32768
  chroot /var/lib/haproxy
  user haproxy
  group haproxy
  daemon
  stats socket /var/lib/haproxy/stats user haproxy group haproxy mode 0640 level admin
  tune.bufsize 32768
  tune.ssl.default-dh-param 2048

  ssl-default-bind-ciphers   ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
  ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384

  ssl-default-bind-ciphersuites   TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384
  ssl-default-server-ciphersuites TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384

  ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 no-tls-tickets
  ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 no-tls-tickets

  nbthread 20
  cpu-map 1/all 0-19

defaults
  log     global
  mode    http
  option  log-health-checks
  option  log-separate-errors
  # option  dontlog-normal
  # option  dontlognull
  option  httplog
  option  splice-auto
  option  socket-stats
  retries 3
  option  redispatch
  maxconn 10000
  timeout connect     5s
  timeout client     50s
  timeout server    450s

listen youdontwantthis
   #
   # HTTP mode. This is bad in the case of step-ca as the step-ca daemon will not see the client cert from the client
   # as such the own protocol for renewing certs will not work.
   #
   bind :8444 tfo ssl crt /etc/ssl/services/letsencrypt/ alpn h2 npn h2 strict-sni
   mode http
   option httplog

   option forwardfor
   #      ^ so we can see the real client IP on the backend

   # you probably want sticky sessions per remote IP so we always talk to the same backend from the same host
   # but for the sake of simplicity of the example we leave that out for now.

   server step-ca 127.0.0.1:8443 ssl ca-file /etc/ssl/ca-bundle.pem verifyhost step-ca-hostname

listen youwantthis
   #
   # plain TCP proxy, haproxy will not touch the packets from the client but we need
   # a way to passthrough the real client IP. see at the end
   #
   bind :8445 tfo
   mode tcp
   option tcplog

   # you probably want sticky sessions per remote IP so we always talk to the same backend from the same host
   # but for the sake of simplicity of the example we leave that out for now.

   server step-ca 127.0.0.1:8443 send-proxy-v2 
   #                             ^ so we can see the real client IP on the backend
   #                               for this we need the proxy protocol support

[maraino]

So it's actually using the transport layer (4) with some metadata in a "header", it is not an layer 7 proxy.

[darix]

correct. and for some business case considerations ... haproxy is not the only proxy which can emit those proxy header. AWS loadbalancer e.g. supports them as well. and there is a k8s ingress controller based on haproxy.

[rmalchow]

this would be a +1 from me. being forced to directly expose the demon seems off. but i think it would be even better to somehow be able to support layer 7 proxying. this would make it possible to only allow public ingress to specific endpoints - rather than all or nothing. this obviously breaks doing the client cert authentication directly in the initial TLS handshake. but haproxy could also do perform the handshake, verify the client cert against the CA, and then pass on the client cert (or a fingerprint thereof) to the step-ca demon in an HTTP header, so this:

frontend client_cert
    bind ::443 ssl crt foobar.pem ca-file ca.pem verify required 
    mode http

    http-request set-header X-SSL-Client-Cert           %[ssl_fc_has_crt]
    http-request set-header X-SSL-Client-Verify         %[ssl_c_verify]
    http-request set-header X-SSL-Client-SHA1           %{+Q}[ssl_c_sha1,hex]
    http-request set-header X-SSL-Client-DN             %{+Q}[ssl_c_s_dn]
    http-request set-header X-SSL-Client-CN             %{+Q}[ssl_c_s_dn(cn)]
    http-request set-header X-SSL-Client-Not-Before     %{+Q}[ssl_c_notbefore]
    http-request set-header X-SSL-Client-Not-After      %{+Q}[ssl_c_notafter]

would yield these headers:

X-SSL: 1
X-SSL-Client-Verify: 0
X-SSL-Client-SHA1: "FF....."
X-SSL-Client-DN: "/C=foo/ST=bar"
X-SSL-Client-CN: "example"
X-SSL-Issuer: "/C=foo/ST=bar/O=smallstep"
X-SSL-Client-Not-Before: "120101100030Z"
X-SSL-Client-Not-After: "160101100030Z"

while at the same time allowing arbitrary proxying (and external inspection of the requests). the question would be wether or not this is good enough to trust - it would be relatively easy to misconfigure this. although misconfiguration on a level to actually make it insecure ... is a bit more difficult.

.rm

[rmalchow]

if one would be thinking about supporting this, then this may need additional config for which proxy's headers we can trust, and expecting a proper client-auth handshake from everyone else?

[darix]

you can do a lot of access control on the tcp proxy already.

[rmalchow]

yes. but, for example, you cannot selectively expose certain endpoints. in my current thinking, i would like to be able to issue tokens, but only allow certificate retrieval (with a token) and renewal for the rest of the world.

[maraino]

I'm also a +1 on supporting the proxy protocol v2. And yes is true that you cannot selectively expose certain endpoints, at least directly, it will be possible to authenticate a client certificate, or perhaps a header. We've been thinking about some solutions where the access needs to be more restricted to the rest of the world. There're some workarounds using other tools, but the main problem is always the bootstrap if we're using certs.

[rmalchow]

this also ties in with https://github.com/smallstep/certificates/discussions/668. the path we were thinking is to issue tokens "internally" (i.e. on a known host). then, it should be possible to use this token for bootstrapping, without any knowledge of the provisioners, right?

so the two paths would be to trust and "external" source (e.g. as described above, with an external component doing the TLS handshake with client authentication) or having an internal configuration that does filtering not only by source ip, but by source ip and target endpoint. i can absolutely see the issues with trusting something external. the second one is a bit better in that respect, but it does require step-ca to see the actual client ip.

another question would be: if one was to configure the entire provider config in a client - why would the /provisioners endpoint be necessary at all? couldn't it simply use the local configuration instead of getting the encrypted key from and endpoint, decrypting it, then signing a JWT? similar things for other provisioners - regardless of wether or not the client_secret needs to be secret - why would it ever be necessary to expose it to an unlimited audience?

as for risks inherent to making certain endpoints public: for JWT, the encrypted key is revealed. this means some sort of PBKDF is involved. this probably means that an attacker can get the key and take brute forcing offline. yes, it might be difficult to guess a proper password - but afaik, this is static, and because you can do it offline, there's not way of limiting the number of attempts made.

for cloud provider IIDs ... there would be at least some level of organizational details exposed, as in "these N accounts all belong to the same entity, so let me try these known credentials on all of them". how you evaluate this type of threat is a matter of personal taste (and circumstances). it might be totally fine, it might be completely unacceptable.

i really like the basic concept of distributing the trust like step-ca does. and it also does tick a lot of other boxes. this issue however, looks rather important to me.

[dopey]

Wanted to follow up here after we had the opportunity to discuss as a team again.

We do want to implement this. @maraino already has some notions for how he wants this implemented. We're currently backed up with other projects, but this is on our roadmap now and we'll figure out prioritization once we've shipped our current work. Will update here once we have some idea of timeline.

If anyone from the community is interested in getting involved, let us know and we can set up a design chat.

[redrac]

@dopey is this still on the roadmap? we are very interested in this feature

[dopey]

Hey @redrac 👋 , thanks for the reminder and I apologize it's taken me a while to reply. To answer the question directly - no, this feature is not currently on the roadmap.

Given our current team size, our roadmap is almost exclusively influenced by product and features requested by customers. We haven't had any customers requesting this feature and therefore it hasn't been prioritized :[

In the mean time, we're happy to accept PRs for this support from the community.

@redrac if you wanna chat about other options, let me know.

[maraino]

It can probably be way simpler, but here is a squid configuration that will allow proxying client connection to the server using the environment variable HTTPS_PROXY=http://proxy:port

http_port 0.0.0.0:8080
    # cache_effective_group proxy
    # cache_effective_user proxy
    cache deny all
    forwarded_for on

    # EOS Proxy peers; no bypassing allowed
    acl localnet src 0.0.0.1-0.255.255.255  # RFC 1122 "this" network (LAN)
    # acl localnet src 10.0.0.0/8             # EOS server net

    acl SSL_ports port 443
    acl SSL_ports port 1-65535

    acl fileupload req_mime_type -i ^multipart/related$

    acl Safe_ports port 80
    acl Safe_ports port 21
    acl Safe_ports port 443
    acl Safe_ports port 70
    acl Safe_ports port 210
    acl Safe_ports port 1025-65535
    acl Safe_ports port 280
    acl Safe_ports port 488
    acl Safe_ports port 591
    acl Safe_ports port 777
    acl CONNECT method CONNECT
    acl all src all
    acl snmpnet src 127.0.0.1
    acl snmppublic snmp_community public

    http_access deny !Safe_ports
    http_access deny CONNECT !SSL_ports
    http_access allow localhost manager
    http_access deny manager
    http_access allow localnet
    http_access allow localhost
    http_access allow fileupload
    http_access allow all

    # Perfomance tweaks
    ipcache_size 10240
    via off
    client_request_buffer_max_size 10250 KB
    connect_timeout 15 minutes
    forward_timeout 15 minutes
    request_timeout 15 minutes
    negative_dns_ttl 5 minutes

    # Hardening
    icp_port 0
    htcp_port 0
    icp_access deny all
    htcp_access deny all

    #
    # Add any of your own refresh_pattern entries above these.
    #
    refresh_pattern ^ftp: 1440  20% 10080
    refresh_pattern ^ftp: 1440  20% 10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|\?) 0 0%  0
    refresh_pattern . 0 20% 4320

A simple way to see this working is running squid, for example, like this:

$ squid -N -f squid.conf

Then we can connect go, for example, to a docker container and run:

$ export HTTPS_PROXY=http://host.docker.internal:8080
$ step ca certificate test.example.com test.crt test.key
$ step ca renew --force test.crt test.key

And if we look at squid logs in /var/logs/access.log or similar you can see entries like:

1715107467.021   3052 127.0.0.1 TCP_TUNNEL/200 5979 CONNECT ca.smallstep.com:9000 - HIER_DIRECT/192.168.1.70 -
1715107485.240      9 127.0.0.1 TCP_TUNNEL/200 5979 CONNECT ca.smallstep.com:9000 - HIER_DIRECT/192.168.1.70 -

The first IP is the source IP, then ca.smallstep.com is the destination and the last ip is the destination IP.

From a machine in my network, my computer running squid and a CA in our SaaS:

$ export HTTPS_PROXY=http://192.168.1.70:8080
$ step ca certificate mariano@smallstep.com mariano.crt mariano.key
✔ Provisioner: mariano@smallstep.com (JWK) [kid: oGiWT3a38vOLu5wAk0NVjspms7kncq7MtxN9zALuhbI]
Please enter the password to decrypt the provisioner key:
✔ CA: https://xxx.yyy.ca.smallstep.com
✔ Certificate: mariano.crt
✔ Private Key: mariano.key

And in the logs:

1715107693.435   6825 192.168.1.72 TCP_TUNNEL/200 3549 CONNECT xxx.yyy.ca.smallstep.com:443 - HIER_DIRECT/35.x.y.z -