search

haproxy / haproxy

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
https://git.haproxy.org/
Other
5.05k stars 801 forks source link

2.0.1 tcp frontend with http backend returns HTTP 408 errors with HTX enabled #150

Closed paradizelost closed 5 years ago

paradizelost commented 5 years ago

Output of haproxy -vv and uname -a

HA-Proxy version 2.0.1 2019/06/26 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.0j  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0j  20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.22 2016-07-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

Linux haproxy 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u3 (2019-06-16) x86_64 GNU/Linux

What's the configuration?


global
#        chroot /var/lib/haproxy
    log 10.0.2.112 local0
    log stderr local0
        daemon
        tune.ssl.default-dh-param 2048

        # Default SSL material locations
        ca-base /etc/haproxy/ssl/certs
        crt-base /etc/haproxy/ssl

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:EECDH+CHACHA20:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES
        ssl-default-server-ciphers TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:TLS13-CHACHA20-POLY1305-SHA256:EECDH+CHACHA20:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

mailers mta
mailer hostname smtp.midco.net:25

defaults
        log     global
#        option  http-server-close
#        option  redispatch
#        option  dontlognull
        timeout connect 50000
        timeout client  50000
        timeout server  50000
#   retries 5

    email-alert mailers mta
       email-alert level alert
       email-alert from haproxy@mydomain.net
       email-alert to danmydomain@gmail.com

#        errorfile 400 /usr/local/etc/haproxy/errors/400.http
#        errorfile 403 /usr/local/etc/haproxy/errors/403.http
#        errorfile 408 /usr/local/etc/haproxy/errors/408.http
#        errorfile 500 /usr/local/etc/haproxy/errors/500.http
#        errorfile 502 /usr/local/etc/haproxy/errors/502.http
#        errorfile 503 /etc/haproxy/errors/503.http
#        errorfile 504 /etc/haproxy/errors/504.http

#frontend www-stats
#        bind 0.0.0.0:8080
#        mode http
#        stats enable
#        stats uri /
#        stats realm Haproxy\ Statistics
#        stats auth <user>:<password>

frontend www-http
        mode http
        bind 0.0.0.0:80
        redirect scheme https

frontend www-https
        mode tcp
    timeout server 8h
    timeout connect 30s
    timeout client 10m  
#   option tcplog
#   option tcpka
#   contimeout 9999
#   clitimeout 99999
#   srvtimeout 99999
        bind 0.0.0.0:443 ssl crt mydomain.net.pem
#        tcp-request inspect-delay 5s
#   timeout client 3h
    http-response set-header Strict-Transport-Security max-age=31356000;\ includeSubdomains;\ preload
    http-response set-header X-Frame-Options DENY
    http-response set-header X-Content-Type-Options nosniff
#        tcp-request content accept if { req_ssl_hello_type 1 }
        #use_backend www-RDP if { ssl_fc_sni -i remote.mydomain.net }
        #use_backend www-RDP if { ssl_fc_sni -i r.mydomain.net }
        use_backend www-guac if { ssl_fc_sni -i guac.mydomain.net }
        use_backend www-nsman if { ssl_fc_sni -i nsman.mydomain.net }
        use_backend www-switch if { ssl_fc_sni -i dellswitch.mydomain.net }
        use_backend www-unifi if { ssl_fc_sni -i unifi.mydomain.net }
        #use_backend www-nsg if { ssl_fc_sni -i ns.mydomain.net }
        use_backend www-apc if { ssl_fc_sni -i battery.mydomain.net }
        use_backend www_ipmiserver if { ssl_fc_sni -i serveripmi.mydomain.net }
        use_backend www_babycam if { ssl_fc_sni -i mydomainbabycam.mydomain.net }
        use_backend www-octoprint if { ssl_fc_sni -i octoprint.mydomain.net }
        use_backend www-splunk if { ssl_fc_sni -i splunk.mydomain.net }
        use_backend www-plex if { ssl_fc_sni -i plex.mydomain.net }
        use_backend www-irc if { ssl_fc_sni -i irc.mydomain.net }
    use_backend www-stats if { ssl_fc_sni -i mystats.mydomain.net }
    use_backend www-ststats if { ssl_fc_sni -i ststats.mydomain.net }
    use_backend www-wd if { ssl_fc_sni -i wd.mydomain.net }
    use_backend www-freenas if { ssl_fc_sni -i freenas.mydomain.net }
    use_backend www-homeassistant if { ssl_fc_sni -i homecontrol.mydomain.net }
    use_backend www-kanboard if { ssl_fc_sni -i kanboard.mydomain.net }
    use_backend www-docs if { ssl_fc_sni -i docs.mydomain.net }
#   use_backend haproxy-test if { ssl_fc_sni -i test-haproxy.mydomain.net }
#   use_backend www-docs-test if { ssl_fc_sni -i testdocs.mydomain.net }

backend www-RDP
        mode tcp
        timeout server 8h
        timeout connect 30s
        server dc 10.0.0.15:443 ssl check verify none

backend www-irc
        mode http
        server irc 10.0.2.21:9000 check

backend www-ststats
        mode http
        server irc 10.0.2.21:5000 check

backend www-stats
        mode http
        server irc 10.0.2.21:8081 check

backend www-nsg
        mode tcp
        timeout server 8h
        timeout connect 30s
        server ns 10.0.2.27:443 check ssl verify none

backend www-octoprint
        mode http
        server octoprint 10.0.3.18:80 check

backend www_ipmiserver
    mode http
    server impiserver 10.0.0.100:443 ssl check verify none

backend www-plex
    mode http
    server plex 10.0.0.122:32400 ssl check verify none

backend www-splunk
    mode http
    server splunk 10.0.2.112:8000 check

backend www_babycam
    mode http
    balance source
    server-template henrycam 3 _http._tcp.henrycam.docker.lan.mydomain.net resolvers int check

backend www-wd
    mode http
    server ovman 10.0.0.102:443 ssl check verify none

backend www-freenas
    mode http
    server freenas 10.0.0.111:80 check

backend www-homeassistant
    mode http
    server homecontrol 10.0.2.21:8123 check send-proxy

backend www-nsman
        mode http
        server netscaler 10.0.2.10:443 ssl check verify none

backend www-apc
        mode http
    option httpchk
        server apc 192.168.1.219:80 

backend www-guac
        mode http
    http-request set-path %[path,regsub(^/,/guacamole/)]
        server guacamole 10.0.2.21:8080 check port 8080 verify none
#   balance source
#   server-template guack-back 3 _http._tcp.guacamole.docker.lan.mydomain.net resolvers int check

backend www-switch
        mode http
        server dellswitch 10.0.1.2:80 check

backend www-unifi
    mode http
    server unifi 10.0.2.23:8443 ssl check verify none

backend www-kanboard
    mode http
    server kanboard 10.0.2.21:88 check

backend www-docs
    mode http
    server mayanedms 10.0.2.21:85 check
#   server mayanedms mayan-edms-app.mayanedms:8000

frontend www-stats
        bind 0.0.0.0:8080
        mode http
        stats enable

        stats uri /
        stats realm Haproxy\ Statistics
        stats auth admin:admin
    stats show-node "${DOCKER_HOST}"

resolvers int
    nameserver fw 10.0.2.1:53

Steps to reproduce the behavior

  1. have tcp front end and http backend with htx setting default (on) and i'm unable to connect to fix either:
  2. change from a tcp front-end to an http one, and disable tcp backends, now works
  3. add "no option http-use-htx" to front end and all back-ends, it now works

Actual behavior

I receive an HTTP 408 error in the browser if i wait long enough, but otherwise page doesn't load

Expected behavior

pages load correctly regardless of frontend/backend configuration, as it did in 1.9.7

Do you have any idea what may have caused this?

started with 2.0.0 release, 2.0.1 did not resolve the issue, config worked correctly on 1.9.7, as disabling htx seems to correc the problem, it seems to be something within htx

Do you have an idea how to solve the issue?

paradizelost commented 5 years ago

I forgot to mention i'm using the docker container

capflam commented 5 years ago

Upgrade from TCP to HTTP in HTX works for me, but without SSL only. And in fact, with an SSL connection the upgrade is done, but then the connection is frozen. I'm investigating.

capflam commented 5 years ago

@cognet pushed a fix. It was also backported to 2.0. See commit 2ab3dada0 for details. Thanks !

paradizelost commented 5 years ago

Thanks for the quick response!