search

haproxy / haproxy

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
https://git.haproxy.org/
Other
4.99k stars 799 forks source link

intermittently req_ssl_sni failing to navigate correct backend with Version 2.8.5 #2515

Open jani4865 opened 7 months ago

jani4865 commented 7 months ago

Detailed Description of the Problem

When I hit report-boutique.example.com intermittenly I am being redirected with 307 to boutique.example.com, Few times it works correctly and I get content from report-boutique.example.com

Issue is with haproxy version 2.8.5 where as same configuration working with 2.6.16

Expected Behavior

At any time req_ssl_sni (report-boutique.example.com) should use backend report

Steps to Reproduce the Behavior

Just use haproxy tag with version and 2.8.5 and use same configuration given below (Please use new browser/incognito window every time )

Do you have any idea what may have caused this?

Not at all

Do you have an idea how to solve the issue?

We Wish

What is your configuration?

frontend https-in
    bind *:443
    tcp-request inspect-delay 5s
    maxconn 2000000
    tcp-request content accept if { req_ssl_hello_type 1 }

   # Explicit acl for - will not use default backend portal to prevent DOS
    acl host_portal req_ssl_sni -i boutique.example.com

    acl host_report req_ssl_sni -i report-boutique.example.com

    use_backend portal if host_portal

    use_backend report if host_report

frontend proxy_services_frontend
    maxconn 50000
    bind *:4443 ssl crt /etc/cert/cert.key no-sslv3 no-tlsv11 accept-proxy
    mode http
    http-after-response set-header Strict-Transport-Security "max-age=31536000"
    acl portal_api_tag path_beg /api /download 
    use_backend portal_api if portal_api_tag
    http-request redirect code 307 location https://boutique.example.com%[path]?%[query] if !portal_api_tag

backend portal
    server portal1 127.0.0.1:4443 send-proxy
    maxconn 200000

backend portal_api
    mode http
    server portal_api1 master-boutique.example.com:8888 check ssl verify none

backend report
    server report1 master-boutique.example.com:7777 check
    maxconn 200000

Output of haproxy -vv

haproxy -vv
HAProxy version 2.8.5-aaba8d0 2023/12/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.5.html
Running on: Linux 6.5.0-1014-aws #14~22.04.1-Ubuntu SMP Thu Feb 15 15:27:06 UTC 2024 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_TFO=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ+SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.0.2zh-fips  30 May 2023
Running on OpenSSL version : OpenSSL 1.0.2zh-fips  30 May 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

uname -a
Linux 7b208737bad8 6.5.0-1014-aws #14~22.04.1-Ubuntu SMP Thu Feb 15 15:27:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Last Outputs and Backtraces

No response

Additional Information

No response

wlallemand commented 7 months ago

Hello,

I can't reproduce your problem. Could you provide logs? That would be difficult to determine what's going on without them. Also you should try to test using a curl command to reproduce.

Also your configuration looks a little bit old fashioned, once you determined the problem you should switch to using ssl_fc_sni directly in a HTTP frontend with an SSL bind, instead of using an intermediate TCP frontend with req_ssl_sni

Regards,

wlallemand commented 5 months ago

Any update about this?

jani4865 commented 6 days ago

I get following logs

Nov  5 11:33:56 haproxy[48]: 114.143.238.10:53768 [05/Nov/2024:11:33:50.871] proxy_services_frontend~ proxy_services_frontend/<NOSRV> -1/-1/+5950 +174 LR 7/3/0/0/0 0/0

Expected logs are 

Nov  5 10:23:31 haproxy[47]: 114.143.238.10:37141 [05/Nov/2024:10:23:31.528] https-in report/report1 2/0/+1 +0 -- 27/25/1/1/0 0/0

This is intermittent

jani4865 commented 6 days ago

@wlallemand Please let me know if you need any other additional info

wlallemand commented 6 days ago

You seem to be analyzing logs from different frontends, all your requests are supposed to get through "https-in" with your configuration, but since this is a TCP frontend there will always be a log for a connection, but a connection could contain multiple requests, and you won't be seeing this in your logs.

It seem like you are using "option logasap" and "option tcplog" but this is not in your configuration snippet.

I recommend that you set "option httplog" or "option httpslog" in your http/https frontend to have more details in your log, so you could see the request and check if the portal_api_tag ACL matched

Since you are choosing a backend for a connection and not for a request, because of mode tcp, if the TCP connection is kept, every other request will be sent to the same backend. You should check in your logs if you have the same IP:port source multiple time in your HTTP "proxy_services_frontend".