search

haproxy / haproxy

HAProxy Load Balancer's development branch (mirror of git.haproxy.org)
https://git.haproxy.org/
Other
4.64k stars 771 forks source link

Unable to update CRL file due to "broken pipe" and "No ongoing transaction!" #2579

Closed akovacs123 closed 1 day ago

akovacs123 commented 1 month ago

Detailed Description of the Problem

I use haproxy version 2.7.12-1~bpo11+1 on Debian Bullseye. I would like to periodically update the CRL file from my CA in order to allow client certificate based authentication with some admin pages. I migrated from version 2.5 where CRL updating worked well.

My CRL is relatively large (22k). I use a cron script to download fresh CRLs from CAs then I locally validate them and create a tmp file. Then I try to update in haproxy. This happens 1-2 times daily. I use below commands to update:

$ echo -e "set ssl crl-file /etc/ssl/haproxy_crl.pem <<\n$(cat /tmp/chain_ca_crl.pem)\n" | socat /run/haproxy/admin.sock -
$ echo "commit ssl crl-file /etc/ssl/haproxy_crl.pem" | socat /run/haproxy/admin.sock -

The first command results in "broken pipe" sometimes (?). Sometimes not.

$ echo -e "set ssl crl-file /etc/ssl/haproxy_crl.pem <<\n$(cat /tmp/chain_ca_crl.pem)\n" | socat /run/haproxy/admin.sock -
2024/05/28 10:46:57 socat[3521098] E write(5, 0x55a59c145000, 5633): Broken pipe

If actually no "broken pipe" is given then commit gives a strange output:

$ echo "commit ssl crl-file /etc/ssl/haproxy_crl.pem" | socat /run/haproxy/admin.sock -
No ongoing transaction! !
Can't commit /etc/ssl/vsq/haproxy_crl.pem!
} 5073
haproxy_server_http_responses_total{proxy="frontend_streaming",server="vik-t-vpfe01b",code="4xx"} 5063
haproxy_server_http_responses_total{proxy="frontend_admin",server="vik-t-vpfe01a",code="4xx"} 0
haproxy_server_http_responses_total{proxy="frontend_admin",server="vik-t-vpfe01b",code="4xx"} 0
(it dumps all haproxy internal variables)

Return code for the above command is 0.

Expected Behavior

Update CRL file properly and provide meaningful error messages if there is an error. Do not dump haproxy variables onto screen.

Steps to Reproduce the Behavior

  1. Use my CRL file, can be downloaded from here: https://www.dropbox.com/scl/fi/m41w9d57bn71zlv1uvc4t/chain_ca_crl.pem?rlkey=slct04o1ait41yqk2sbmk9r8i&dl=0
  2. Use below provided haproxy config snippet
  3. Use above commands to update CRL

Do you have any idea what may have caused this?

It worked in version 2.5 with no problem, so something changed in 2.7

Do you have an idea how to solve the issue?

No response

What is your configuration?

frontend www_https_in
  bind x.y.z.w:80
  bind x.y.z.w:443 ssl crt /etc/ssl/mycert_chain.pem ecdhe secp384r1 ca-file /etc/ssl/vsq/client_cert_chain.pem ca-verify-file /etc/ssl/client_cert_chain.pem verify optional crt-ignore-err 10,12,23 crl-file /etc/ssl/haproxy_crl.pem
  # HSTS
  http-response set-header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;"
  http-request set-header X-Forwarded-Proto https

  # Redirect to HTTPS
  redirect scheme https code 301 if !{ ssl_fc }

  default_backend frontend

Output of haproxy -vv

# haproxy -vv
HAProxy version 2.7.12-1~bpo11+1 2024/04/06 - https://haproxy.org/
Status: End of life - please upgrade to branch 2.8.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.12.html
Running on: Linux 5.10.0-29-amd64 #1 SMP Debian 5.10.216-1 (2024-05-03) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1w  11 Sep 2023
Running on OpenSSL version : OpenSSL 1.1.1w  11 Sep 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

No response

git001 commented 1 month ago

Because of this line Status: End of life - please upgrade to branch 2.8. any chance to use a more recent version and a not EOL Version?

akovacs123 commented 1 month ago

Because of this line Status: End of life - please upgrade to branch 2.8. any chance to use a more recent version and a not EOL Version?

Thanks for the feedback. I upgraded to haproxy 2.8.9 and experience the same. However, haproxy variablea are not printed anymore and output slightly changed

# echo -e "set ssl crl-file /etc/ssl/haproxy_crl.pem <<\n$(cat /tmp/chain_ca_crl.pem)\n" | socat /run/haproxy/admin.sock -
2024/05/28 13:38:12 socat[3583528] E write(5, 0x557433106000, 5633): Broken pipe
# echo "commit ssl crl-file /etc/ssl/haproxy_crl.pem" | socat /run/haproxy/admin.sock -
No ongoing transaction! !
Can't commit /etc/ssl/haproxy_crl.pem!

Haproxy info

HAProxy version 2.8.9-1~bpo11+1 2024/04/06 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.9.html
Running on: Linux 5.10.0-29-amd64 #1 SMP Debian 5.10.216-1 (2024-05-03) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1w  11 Sep 2023
Running on OpenSSL version : OpenSSL 1.1.1w  11 Sep 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Darlelet commented 1 month ago

cli commands use general purpose buffers that are limited in size to tune.bufsize (defaults to 16k)

With a CLR file > tune.bufsize the command will fail. Before 498520fdf505f05cde2f99c70c4e98fe90098961 no error would be emitted which is quite error-prone, but with latest versions (the fix was marked for backports on all stable versions) the user will be warned when such case occur.

I see that you already upgraded to 2.8.9 which is missing 498520fdf505f05cde2f99c70c4e98fe90098961 unfortunately (the fix will be included in 2.8.10).

To circumvent the buffer size limitation, either try to increase tune.bufsize (please note that this will have performance implications for http processing), or try to split your CRL file into multiple (smallers) chunks if that's an option.

Darlelet commented 1 month ago

See #2324

akovacs123 commented 1 month ago

Thanks for your response. After increasing tune.bufsize it is fine. Referred ticket already addresses this to be covered by a feature request. It would be really useful to make tune.bufsize independent of this (it seems the two things have nothing to do with each other: http processing and CLI CRL update).

If I may recommend to consider adjusting of return codes. Even if CRL set command failed due to lack of buffer area haproxy could return an error. I use a custom bash script to update CRL and this way it is impossible to detect that something has gone wrong.

Thanks.

Darlelet commented 1 month ago

If I may recommend to consider adjusting of return codes. Even if CRL set command failed due to lack of buffer area haproxy could return an error. I use a custom bash script to update CRL and this way it is impossible to detect that something has gone wrong.

That's already the case with 498520fdf505f05cde2f99c70c4e98fe90098961, the following error message is emitted once the buffer size is reached: 

The command is too big for the buffer size. Please change tune.bufsize in the configuration to use a bigger command.

(The fix will be available in the upcoming 2.8.10, and perhaps in 2.6 and 2.4 as it is specified that it "could" be backported to all stable versions)