Tcp-request reject not working on docker version>1.5.16

[gregbkr]

Hello, After many tests, seems like I can't reject any tcp-connection via haproxy.cfg with official docker image tag > 1.5.16:

tcp-request connection reject if { src_conn_rate(Abuse) ge 2 }     # 10 conn in 3 sec
tcp-request connection reject if { src_conn_cur(Abuse) ge 2 }      # 10 concurent conn

Working 1.5.16:

On Server mylb:

docker run --name mylb -v /root/env/myhaproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -p "80:80" -p "8881:8881" -it -d haproxy:1.5.16 haproxy -db -f /usr/local/etc/haproxy/haproxy.cfg 

On a client:

ab -n 3 -c 3 http://ip_mylb:80/
Benchmarking 185.x.x.x (be patient)...apr_socket_recv: Connection reset by peer (104) --> block fine!

ab -n 2 -c 1 http://ip_mylb:80/
--> connection success

Not working: 1.6

Server mylb:

docker run --name mylb -v /root/env/myhaproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro -p "80:80" -p "8881:8881" -it -d haproxy:1.6 haproxy -db -f /usr/local/etc/haproxy/haproxy.cfg 

On a client:

ab -n 300 -c 300 http://ip_mylb:80/
--> connection success --> not blocking anything. 

I can monitor the Abuse tick-table and see these open connections via

docker exec -it mylb apt-get update
docker exec -it mylb apt-get install socat
docker exec -it mylb bash -c 'echo "show table Abuse" | socat unix:/var/run/haproxy.stats -'

Seems that only the blocking conf do not block connection.

Same config in both tests:

myhaproxy.cfg:

global
    maxconn 4096 # count about 1 GB per 20 000 connections
    user nobody
    group nogroup
    chroot  /tmp
    ulimit-n 65536

    # haproxy stats socket is available at /var/run/haproxy.stats --> echo "show table Abuse" | socat unix:/var/run/haproxy.stats -
    stats socket /var/run/haproxy.stats mode 600 level admin
    stats timeout 2m

    # Security config, cf: https://wiki.mozilla.org/Security/Server_Side_TLS#Recommended_configurations
    tune.ssl.default-dh-param 2048
    ssl-default-bind-options no-sslv3 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES128-SHA:AES256-SHA256:AES256-SHA:SRP-RSA-3DES-EDE-CBC-SHA:DES-CBC3-SHA

    # lower your record size to improve Time to First Byte (TTFB) - see: https://plus.google.com/+IlyaGrigorik/posts/Q5kgmKve54p
    tune.ssl.maxrecord 1370

defaults
    maxconn 4000 # Must be slight smaller that global, so you can still connect
    log     global
    mode    http
    option  httplog
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms
    option  dontlognull
    option  splice-auto
    option  http-keep-alive
    option  redispatch
    retries 3
    # disconnect slow handshake clients early, protect from resources exhaustion attacks
    timeout http-request    5s #Slowloris protection
    timeout connect         5s
    timeout queue           1m
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    # errorfile 500 /etc/haproxy/errors/500.http

listen stats
    mode http
    log global
    bind *:8881
    maxconn 10
    stats enable
    stats hide-version
    stats refresh 30s
    stats show-node
    #stats auth user:pass
    stats uri /

backend Abuse
    stick-table type ip size 1m expire 30m store conn_rate(3s),conn_cur,gpc0,http_req_rate(10s),http_err_rate(20s)

frontend http-in
    mode http
    option httplog
    bind *:80
    timeout client 10
    rspadd Strict-Transport-Security:\ max-age=15768000

    ###############################################
    ## TCP protect against DOS, SCANNING, etc
    #tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }     # whitelist
    #tcp-request content reject if { src -f /etc/haproxy/blacklist.lst }        # blacklist
    tcp-request connection reject if { src_conn_rate(Abuse) ge 2 }     # 10 conn in 3 sec
    tcp-request connection reject if { src_conn_cur(Abuse) ge 2 }      # 10 concurent conn
    tcp-request connection track-sc1 src table Abuse    # record them in table Abuse
    ################################################

    default_backend www_backend

backend www_backend
    option forwardfor # add the X-Forwarded-For header
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    server myserver 185.x.x.x:80 maxconn 32

I probably missed something obvious.

Thank you for your feedback! God day! Greg.

[gregbkr]

Seems that image 1.6.3 now rejecting well my request. But 1.6.4 still broken... Thx for any updates...

[tianon]

Sorry for the massive delay -- it'd be great if we could reproduce this with haproxy outside of a container.

We don't really do anything terribly custom in building the image, so it should be a pretty stock haproxy setup, and I'm thus going to close. If we can do more debugging and determine that it's not simply a configuration issue and is an issue with the image, I'd love to get to the bottom of it and fix whatever we've overlooked.