FCGI_STDIN record not sent with empty POST/PUT/PATCH over HTTP/2

Detailed Description of the Problem

When doing a POST, PUT or PATCH request without a payload over HTTP/2, HAProxy does not send the empty FCGI_STDIN record causing some FCGI servers to wait indefinitely to start processing the request, resulting in a gateway timeout and risking DOS.

HAProxy does send the empty STDIN record properly when doing, for example:

GET over HTTP/2
POST with at least 1 byte of payload over HTTP/2
POST over HTTP/1.1
GET over HTTP/1.1

Here are pcap dumps of some of the above requests (zipped because pcap files aren't accepted): pcap-dumps.zip

A screenshot for reference of the POST over HTTP/1.1:

A screenshot for reference of the POST over HTTP/2: Note that here, in line 4, the FCGI_STDIN is not sent.

There may be other request methods that have the same problem. It is probable that this is also a problem with HTTP/3 although I have been unable to test this.

Expected Behavior

Have HAProxy constently send the empty FCGI STDIN record indicating the end of the request data, regardless of the HTTP version used to make the request at the frontend or the request method used (GET, POST, PUT, PATCH etc).

Steps to Reproduce the Behavior

These steps show how to capture the fcgi requests using wireshark or tcpdump. Note that when using this method, the fcgi app will respond because it uses libfcgi which seems to start processing when the empty FCGI_PARAMS record is received. However, it does demonstrate that the FCGI_STDIN is not sent.

run haproxy with the given configuration or similar, with a proper SSL certificate chain file configured as required for HTTP/2.
prepare a fastcgi app (optional):

on Debian, as root: apt-get install libfcgi-dev spawn-fcgi

Then create fcgiapp.c:

#include <fcgi_stdio.h>

int main() {
    while (FCGI_Accept() >= 0)
        printf("Content-type: text/html\r\n\r\nHello World\n");
}

and compile with gcc fcgiapp.c -lfcgi -o fcgiapp

run the fastcgi app: spawn-fcgi -n -p 9000 ./fcgiapp alternatively, because we're only capturing packets, you could skip step 2 and simply run netcat: nc -l -p 9000 > /dev/null
run wireshark or tcpdump, for example tcpdump -i lo port 9000 -X --print -w /tmp/dump.pcap
make an empty POST request over HTTP/2: curl -v -k -d "" --http2 https://127.0.0.1/
Observe the missing FCGI_STDIN in the capture.

Do you have any idea what may have caused this?

No response

Do you have an idea how to solve the issue?

I don't have a solution but I do have a pointer in the right direction.

I added the following to the haproxy.conf:

global
    # https://www.haproxy.com/documentation/haproxy-runtime-api/reference/trace/
    expose-experimental-directives
    trace fcgi event +any
    trace fcgi level developer
    trace fcgi verbosity simple
    trace fcgi sink stderr
    trace fcgi start now

then did two requests: curl -v -k -d "" --http1.1 https://127.0.0.1/ and curl -v -k -d "" --http2 https://127.0.0.1/ (part of these logs is attached) haproxy-bug-report-http1.txt haproxy-bug-report-http2.txt

and compared the logs. The following section is present in the http1.1 log but not in the http2 log:

[00|fcgi|2|x_fcgi.c:4023] sending FCGI STDIN record [rsp:MSG_RPBEFORE] - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000304)
#011 htx=0x7f114c44f680(size=16336,data=1,used=1,wrap=NO,flags=0x00000010,extra=0,first=6,head=6,tail=6,tail_addr=137,head_addr=0,end_addr=136)
[00|fcgi|5|x_fcgi.c:1790] fcgi_strm_send_empty_stdin(): in [rsp:MSG_RPBEFORE] - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000304)
[00|fcgi|5|x_fcgi.c:1720] fcgi_strm_send_empty_record(): entering [rsp:MSG_RPBEFORE] - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000304)
[00|fcgi|5|x_fcgi.c:1754] fcgi_strm_send_empty_record(): leaving [rsp:MSG_RPBEFORE] - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000304)
[00|fcgi|2|x_fcgi.c:1794] FCGI STDIN record xferred [rsp:MSG_RPBEFORE] - VAL=0 - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000306)
[00|fcgi|1|x_fcgi.c:1795] FCGI request fully xferred [rsp:MSG_RPBEFORE] - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000306)
[00|fcgi|3|x_fcgi.c:1796] stdin data fully sent [rsp:MSG_RPBEFORE] - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000306)

which tells me that this line of code: https://github.com/haproxy/haproxy/blob/v2.9.0/src/mux_fcgi.c#L4023 is not executed. Both logs show this line right before the above snippet:

[00|fcgi|2|x_fcgi.c:1778] FCGI PARAMS record xferred [rsp:MSG_RPBEFORE] - VAL=0 - fconn=0x7f114cab8600(RDH,0x00001000) fstrm=0x7f114cb5ae40(1,OPN,0x00000304)

so ret must be != 0 (because this line in fcgi_strm_send_empty_params is executed) and the goto on line 4020 is not executed. That can only mean that the if (htx_is_unique_blk(htx, blk) && (htx->flags & HTX_FL_EOM)) on line 4022 evaluated to false.

What is your configuration?

defaults
    mode http
    timeout connect 10s
    timeout client  20s
    timeout server  20s

frontend fe_http
    bind 127.0.0.1:443 ssl crt /etc/ssl/private/haproxy.pem alpn h2,http/1.1
    default_backend be_default

backend be_default
    server s0 127.0.0.1:9000 proto fcgi
    use-fcgi-app my-fcgi-app

fcgi-app my-fcgi-app
    docroot /

Output of `haproxy -vv`

HAProxy version 2.9.6-1~bpo12+1 2024/03/01 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.6.html
Running on: Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=1).
Built with OpenSSL version : OpenSSL 3.0.11 19 Sep 2023
Running on OpenSSL version : OpenSSL 3.0.11 19 Sep 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [  OT] opentracing
        [SPOE] spoe
        [TRACE] trace

Last Outputs and Backtraces

No response

Additional Information

Debian Bookworm, using the apt repository http://haproxy.debian.net bookworm-backports-2.9 main from the wizard at https://haproxy.debian.net/ .

I encountered this problem while working on my own fastcgi server implementation which depends on the empty FCGI_STDIN record being sent for each request to initiate processing the request. There are of course workarounds possible, such as checking the content length request header and start processing upon receiving the empty FCGI_PARAMS record. However, this complicates things. The fact that the record is supposed to be sent for every request according to the FCGI specification, that it is sent for GET requests which have no body payload and that it is sent when using HTTP/1.1 led me to file this bug report.

haproxy / haproxy