stalling uploads for proxy over LAN, cannot set proxy_protocol timeout

[blob42]

Hi. My uploads stall when using docker-proxy. I have had this issue since years but did not bother to track the problem before. I have exactly the same issue with a self hosted nextcloud instance, I cannot upload files.

So the issue is with uploads which randomly stall, if I upload directly to the upstream server without proxy I have no issue. I tried many options and issues and this one seems close enough.

I would like to set the option for caddy proxy_protocol timeout from this config:

servers :443 {
        listener_wrappers {
            proxy_protocol {
                timeout 0
            }
        }
    }

Or any other options for the :443 server.

I tried adding to pre loaded Caddyfile but I guess this is overwritten by docker-proxy. I have the following in my compose file where I define caddy-docker-proxy:

 labels:
      caddy_0: ""
      caddy_0.cert_issuer: internal
      caddy_0.log.level: "DEBUG"

      caddy_1: ":443"
      caddy_1.listener_wrappers.proxy_protocol.timeout: 0

Which gives the following error:

caddy-c-gopher  | {"level":"info","ts":1708455542.363155,"logger":"docker-proxy","msg":"Process Caddyfile","logs":"[ERROR]  Removing invalid block: Caddyfile:46: unrecognized directive: listener_wrappers\n:443 {\n\tlistener_wrappers {\n\t\tproxy_protocol {\n\t\t\ttimeout 0\n\t\t}\n\t}\n}\n\n"}

More context:

Looking at the debug logs of caddy-docker-proxy, there is some latency then the server reloads the full config after a few seconds while the upload request keeps stalled.

It is important to note that this caddy-proxy instance is running within a LAN and the upload request comes from said LAN (vlan). I have an other proxy instance over a public IP without any issues with uploads.

Here is a simple diagram of both setups:

Failing setup:
<PC> --=>upload request=>--(LAN)----(ROUTER)----(PROXY)----(CONTAINER)

working setup:
<PC> --=>upload request=>--(INTERNET)--->---(VPN)----(PROXY)----(CONTAINER)

Example upload request with log

• Request:

  
  (130) curl -vv -L  -H "Content-Type: multipart/form-data" -F file="@gb1.ogg" -F model="whisper-1" http://
  v1/audio/transcriptions

• Host localai.c.gopher:80 was resolved.

• IPv6: (none)

• IPv4: 10.50.96.80

• Trying 10.50.96.80:80...

• Connected to localai.c.gopher (10.50.96.80) port 80

  POST /v1/audio/transcriptions HTTP/1.1 Host: localai.c.gopher User-Agent: curl/8.5.0 Accept: / Content-Length: 1667982 Content-Type: multipart/form-data; boundary=------------------------uUYF81ktab9sihxPYyQV4U Expect: 100-continue

  < HTTP/1.1 100 Continue < HTTP/1.1 100 Continue < Server: Caddy

• caddy-docker-proxy logs:

  caddy-c-gopher  | {"level":"debug","ts":1708460149.4224715,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"10.50.96.9:8080","total_upstreams":1}
  ..... CONFIG RELOADED LOG ....
  caddy-c-gopher  | {"level":"info","ts":1708460162.396117,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
  caddy-c-gopher  | {"level":"debug","ts":1708460169.8951006,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.50.96.9:8080","duration":184.091638161,"request":{"remote_ip":"10.40.1.1","remote_port":"46738","client_ip":"10.40.1.1","proto":"HTTP/1.1","method":"POST","host":"localai.c.gopher","uri":"/v1/audio/transcriptions","headers":{"X-Forwarded-Host":["localai.c.gopher"],"Content-Type":["multipart/form-data; boundary=------------------------uUYF81ktab9sihxPYyQV4U"],"Expect":["100-continue"],"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"Content-Length":["1667982"],"X-Forwarded-For":["10.40.1.1"],"X-Forwarded-Proto":["http"]}},"error":"context canceled"}
  caddy-c-gopher  | {"level":"info","ts":1708460169.8951473,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"10.40.1.1","remote_port":"46738","client_ip":"10.40.1.1","proto":"HTTP/1.1","method":"POST","host":"localai.c.gopher","uri":"/v1/audio/transcriptions","headers":{"User-Agent":["curl/8.5.0"],"Accept":["*/*"],"Content-Length":["1667982"],"Content-Type":["multipart/form-data; boundary=------------------------uUYF81ktab9sihxPYyQV4U"],"Expect":["100-continue"]}},"bytes_read":66608,"user_id":"","duration":184.09177444,"size":0,"status":100,"resp_headers":{"Server":["Caddy"]}}

[francislavoie]

servers is a global option, you don't apply it to a site.

Are you actually using proxy protocol? Do you have another proxy in front of Caddy which adds it? If not then that's definitely not the right thing to configure.

[blob42]

How can I setup the global servers options ?

Sorry you're right I don't have an other proxy in front.

I donty know if I should report this on the Caddy project ? I will take some pcap captures as well.

[francislavoie]

You're already configuring global options with your caddy_0 labels.

Configuring proxy_protocol here is definitely wrong then. That's specifically only needed if you have another proxy in front of Caddy which prepends some data to the traffic sent along which contains the original IP address. The timeout is for "reading the proxy protocol bytes". Doesn't make sense for you.

From the log message, looks like only "bytes_read":66608 but the request was "Content-Length":["1667982"]. It took 180 seconds to read only 66KB out of 1.6MB. That's strange. It just seems to me like your network is really really slow. Configuring timeouts won't help with that.

[blob42]

@francislavoie thanks for the detailed answer.

I am suspecting some kind of congestion problem on my network. It doesn't make sense that it works well if I remove the caddy proxy in my LAN.

I still would like to know how to configure the global :443 and :80 server options using labels if possible.

I will close the ticket for now as the issue is not likely with this project.

Thanks again for your time.