search

haproxy / haproxy

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
https://git.haproxy.org/
Other
4.89k stars 790 forks source link

option transparent is not working #2233

Closed InputOutputZ closed 7 months ago

InputOutputZ commented 1 year ago

Detailed Description of the Problem

I have tried several recent HAProxy releases up to 2.9-dev2 to get option transparent working and I couldn't. After configuring all the required elements to get transparent proxy working, and establishing connection to the transparent proxied backend, HAProxy doesn't show any connection was established, and it stalls the connection without any response, if I force quit the connection from terminal, in the logs I get the following:-

Jul 27 17:52:18 localhost haproxy[22826]: MYLOCALIP:60598 [27/Jul/2023:17:52:17.529] ft_redis~ bk_redis/<NOSRV> 31/-1/1371 0 CC 1/1/0/0/0 0/0
Jul 27 17:52:18 server haproxy: <134>Jul 27 17:52:18 haproxy[22826]: MYLOCALIP:60598 [27/Jul/2023:17:52:17.529] ft_redis~ bk_redis/<NOSRV> 31/-1/1371 0 CC 1/1/0/0/0 0/0

I spent whole day trying to get transparent working and so far I couldn't, I wonder if it's a bug?

Expected Behavior

Establishing successful connection to my redis cluster using client ip address instead of haproxy server ip address.

Steps to Reproduce the Behavior

1- Install and configure tls-client-authenticated redis cluster. 2- Configure HAProxy with at least the following settings for redis cluster.

frontend ft_redis
        mode tcp
        bind :7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
        bind [::]:7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
        use_backend bk_redis

backend bk_redis
        option transparent
        source 0.0.0.0 usesrc clientip
        server redis :6379 ssl verify required crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem

Do you have any idea what may have caused this?

I wonder if it's iptables issue but I tried following official haproxy tutorial alongside other tutorials iptables rules but the problem still persists.

-1 https://www.haproxy.com/blog/howto-transparent-proxying-and-binding-with-haproxy-and-aloha-load-balancer -2 https://www.loadbalancer.org/blog/configure-haproxy-with-tproxy-kernel-for-full-transparent-proxy/

Do you have an idea how to solve the issue?

No

What is your configuration?

global
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:TLS_AES_128_CCM_8_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
        ssl-default-server-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:TLS_AES_128_CCM_SHA256:TLS_AES_128_CCM_8_SHA256
        ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-dh-param-file /etc/haproxy/dhparams.pem
        log stderr local0 info
        log 127.0.0.1 local2 notice
        log 127.0.0.1 local3
        chroot /var/empty
        #user haproxy
        #group haproxy
        stats socket /var/run/haproxy.sock mode 660 level admin
        stats timeout 10s
        tune.bufsize 16384000
        h2-workaround-bogus-websocket-clients
        tune.h2.header-table-size 65536
        tune.h2.initial-window-size 3048576
        tune.h2.max-concurrent-streams 500

defaults
    log global
    option httplog
    option http-keep-alive
    option abortonclose
    option redispatch
    retries 5
    timeout client 1h
    timeout connect 1h
    timeout server 1h
    timeout tunnel 1h
    timeout queue 1h
    timeout client-fin 1s
    timeout server-fin 1s
    timeout http-keep-alive 2s

frontend ft_redis
        mode tcp
        bind :7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
        bind [::]:7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
        use_backend bk_redis

backend bk_redis
        option transparent
        source 0.0.0.0 usesrc clientip
        server redis :6379 ssl verify required crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem

Output of haproxy -vv

HAProxy version 2.9-dev2 2023/07/21 - https://haproxy.org/
Status: development branch - not safe for use in production.
Known bugs: https://github.com/haproxy/haproxy/issues?q=is:issue+is:open
Running on: Linux 6.4.4-1.el7.elrepo.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jul 19 17:42:42 EDT 2023 x86_64
Build options :
  TARGET  = generic
  CPU     = generic
  CC      = cc
  CFLAGS  = -std=gnu99
  OPTIONS = USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_CRYPT_H=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_SYSTEMD=1 USE_PCRE=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM -CPU_AFFINITY +CRYPT_H -DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS -GETADDRINFO -KQUEUE -LIBATOMIC +LIB
CRYPT -LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO -THREAD -THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 3.1.0 14 Mar 2023
Running on OpenSSL version : OpenSSL 3.1.0 14 Mar 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.3.0
Built without multi-threading support (USE_THREAD not set).
Built with zlib version : 1.2.11
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 4.8.5 20150623 (Red Hat 4.8.5-44)

Available polling systems :
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 2 (2 usable), will use poll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : none

Available filters :
    [BWLIM] bwlim-in
    [BWLIM] bwlim-out
    [CACHE] cache
    [COMP] compression
    [FCGI] fcgi-app
    [SPOE] spoe
    [TRACE] trace

Last Outputs and Backtraces

Jul 27 17:52:18 localhost haproxy[22826]: MYLOCALIP:60598 [27/Jul/2023:17:52:17.529] ft_redis~ bk_redis/<NOSRV> 31/-1/1371 0 CC 1/1/0/0/0 0/0
Jul 27 17:52:18 server haproxy: <134>Jul 27 17:52:18 haproxy[22826]: MYLOCALIP:60598 [27/Jul/2023:17:52:17.529] ft_redis~ bk_redis/<NOSRV> 31/-1/1371 0 CC 1/1/0/0/0 0/0

Additional Information

RHEL7, CSF Firewall.

InputOutputZ commented 1 year ago

By the way, when I remove the following two lines

        option transparent
        source 0.0.0.0 usesrc clientip

I'm can establish successful connection to redis cluster with tls cluster, replication and client authentication enabled yet client one is left being bypassed and won't be verified, due to the IP haproxy sends, which is haproxy server IP and as a result Redis Cluster won't recognise the connection as client/remote one but local and ignores verifying client ssl certificate.

Darlelet commented 1 year ago

Hi,

transparent option will make haproxy use the original destination address for sessions without cookies

(See: https://docs.haproxy.org/dev/configuration.html#4.2-option%20transparent)

But it will not make haproxy use the client source address as source address for haproxy->server connections. So it might not be what you're looking for.

On the other hand, source with usesrc clientip could indeed help you achieve this. Also you might be interested in the transparent bind option to virtually bind haproxy on the destination server IP address as well, so that clients think that they directly speak with the redis server.

Note that source 0.0.0.0 usesrc clientip will usually require NAT rules on the destination server so that server generated packets can be routed back to haproxy which will then take care of answering to the clients (server will see packets as if they came directly from the client ip address, not haproxy network address, so without the appropriate rules packets may be lost on the server, or could be directly forwarded to the clients, bypassing haproxy for server->client reverse path, which could result in network errors and in your case will prevent your setup from working properly)

If you still encounter issues, I would suggest that you perform a network capture on the redis host to ensure that haproxy's packets are properly reaching your server, and then that the server knows how to respond to them.

InputOutputZ commented 1 year ago

Hi Darlelet,

Thanks for your elaboration.

On the other hand, source with usesrc clientip could indeed help you achieve this.
Also you might be interested in the transparent [bind option](https://docs.haproxy.org/dev/configuration.html#5.1-transparent) to virtually bind haproxy on the destination server IP address as well, so that clients think that they directly speak with the redis server.

I think this is what I'm actually doing, if you look here, I listen on 7618 i.e. destination server, and proxy directly to redis server at 6379, for now both hosted in the same server.

frontend ft_redis
        mode tcp
        bind :7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
        bind [::]:7618 tfo ssl crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem
        use_backend bk_redis

backend bk_redis
        option transparent
        source 0.0.0.0 usesrc clientip
        server redis :6379 ssl verify required crt /etc/redis/cert/redisfull.pem ca-file /etc/redis/cert/ca.pem

If I remove the following two lines

        option transparent
        source 0.0.0.0 usesrc clientip

I can establish successful remote connection to the redis server using haproxy port 7618 i.e. haproxy packets are reaching redis server and client think as if they directly speak with redis server, however with option transparent something is going wrong and still couldn't figure it out, somehow I feel the packets are lost and are not routed properly with transparent option and I think it's something to do with NAT rules as you mentioned since I confirmed all configurations are set properly except ip table rules, having I applied following IP tables rules and expect it to route the packets to the point where it meant to go yet I couldn't find way to analyse packets which haproxy receives, and how to trace what happens to it when it passes through iptables and why do I get bk_redis/:-

iptables -t mangle -N DIVERT iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT iptables -t mangle -A DIVERT -j MARK --set-mark 1 iptables -t mangle -A DIVERT -j ACCEPT

If you still encounter issues, I would suggest that you perform a network capture on the redis host to ensure that haproxy's packets are properly reaching your server, and then that the server knows how to respond to them.

I know how to perform client list command in haproxy, also I can see connection established with transparent yet with I get noserv response, and wonder if my server requires any extra. specific ip tables rules?

Also, most of the examples on how to configure tproxy I found for http mode usually web servers like nginx and httpd and I wonder if there is anything different I have to adjust for tcp mode?

Any further inputs would be very much appreciated with thanks.

Zakaria.

Darlelet commented 1 year ago

To check that packets are reaching the redis host (with or without NAT rules), you could use tcpdump to capture network traffic on the proper interface.

In addition to the NAT rules you mentioned, that seem to "only" mark the relevant packets with 0x1, you would need to configure routing rules on the host so that marked packets are routed through the correct gateway: here haproxy node.

Unfortunately I won't be able to further help here, Github issues are meant to be used for bug reports or features requests only, yet it's seems that your issue lies in configuration / environment problems and not in haproxy software directly.

I advise you to ask your questions on the haproxy mailing list (haproxy@formilux.org) or on the community forum to seek for help and feedbacks from experienced haproxy users.

But don't hesitate to come back on this thread if you still suspect that haproxy doesn't behave like it should and thus you think you found an actual bug in the software or issue in the documentation.

Thanks

capflam commented 7 months ago

I'm closing the issue. Thanks !