ERR_QUIC_PROTOCOL_ERROR with caddy 2.7 and api-platform docker

[Abel-moi]

I'm using api-platform docker with caddy 2.7 because of a problem of compilation with go 1.20. Using caddy 2.7 is supposed to be a temporary fix but since this modification, i have an error ERR_QUIC_PROTOCOL_ERROR in my caddy's logs and i see nothing about this error and i don't know how to fix it.

Thanks for your answers.

[francislavoie]

Thanks for opening an issue! We'll look into this.

It's not immediately clear to us what is going on, so we'll need your help to understand it better.

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! It will ask for some information you've already provided; that's OK, just fill it out the best you can. :+1:

I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```

### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```

### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```

## 2. Description

### 2a. What happens (briefly explain what is wrong)

### 2b. Why it's a bug (if it's not obvious)

### 2c. Log output

```
paste terminal output or logs here
```

### 2d. Workaround(s)

### 2e. Relevant links

## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file: ``` { ... } ``` Open terminal and run Caddy: ``` $ caddy ... ``` Make an HTTP request: ``` $ curl ... ``` Notice that the result is ___ but it should be ___.

[Abel-moi]

I'm using Docker with usually caddy 2.6.4, but with after this issue: https://github.com/api-platform/api-platform/issues/2452

i had to change that :

FROM caddy:${CADDY_VERSION}-builder-alpine AS api_platform_caddy_builder

RUN xcaddy build v2.6.4 \
    --with github.com/dunglas/mercure \
    --with github.com/dunglas/mercure/caddy \
    --with github.com/dunglas/vulcain \
    --with github.com/dunglas/vulcain/caddy

Since then, some of my users started to have ERR_QUIC_PROTOCOL_ERROR in their browsers when they use my web app.

I don't know how to reproduce the error on my computer but for the users that have the problem, the only workaround i found is to disable quic protocol in chrome://flags. For now, the users all use windows 10 or 11, we don't have the same problem with linux. But not all the windows users have this problem.

[francislavoie]

Again, without full context, we can't help. Please fill out all the requested information. If we have to make assumptions because you didn't tell us specifics, it'll waste time both for you and for us.

[mholt]

Full logs would at least be helpful.

ERR_QUIC_PROTOCOL_ERROR in my caddy's logs

This is an error that browsers show, I don't think this would appear in your Caddy logs. Can you provide the full logs along with an example request, etc?

[Abel-moi]

yes i made a mistake it's not in caddy logs, but in browser sorry.

[mholt]

So what's in the Caddy logs?? (Enable debug mode please)

[simon-fa]

Hi,

This issue is really strange. We get this error with some computers, not all, mostly when using Chrome browser…

Most users don't have this problem. For those that have this issue, we can go to chrome://flags, then disable the Quic protocol to instantly make it works.

We are going to enable the debug mode on Monday to try to get more info. As for now, I can't see any strange logs in Caddy…

[simon-fa]

Again today...

[francislavoie]

The Chrome source code isn't too helpful to explain what that error code means: https://chromium.googlesource.com/external/github.com/google/proto-quic/+/merge-to-59.0.3043.0/src/net/quic/core/quic_error_codes.h#181 I'm not sure what RTOs are.

[mholt]

FWIW I've seen random "QUIC_PROTOCOL_ERROR" errors from Chrome on other sites that don't use Caddy. :man_shrugging: It could just be a bug in Chrome too.

Will be hard to narrow down without other HTTP/3 clients. I'd recommend curl (with HTTP/3 compiled in).

[simon-fa]

It happened once with Firefox too, but most of our users use Chrome.

But more and more users have the same issue...

[mholt]

The output of curl -v would be the most helpful, along with debug logs from Caddy during the request.

[simon-fa]

We activated the debug mode and are waiting for a user with the same issue

[simon-fa]

I got these info :

{"level":"info","ts":1687935837.0303285,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details."}

Could it generate the Chrome error ?

[mholt]

Unlikely. That's just a notice that your OS settings aren't tuned for optimal QUIC performance.

[simon-fa]

No further entries in the logs… And the problem seems to have vanished since we upgraded to beta 2.

[mholt]

Awesome. Thanks for the update!

[simon-fa]

It's back...

{"level":"error","ts":1689263640.0222597,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263640.041858,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263640.0447729,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263640.0470617,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263652.2237778,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000139996,"size":27,"status":400,"resp_headers":{"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"],"Content-Security-Policy":["default-src 'self'"],"X-Frame-Options":["DENY"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}
{"level":"error","ts":1689263652.2484465,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263652.5164459,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"],"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000120985,"size":27,"status":400,"resp_headers":{"Content-Security-Policy":["default-src 'self'"],"X-Frame-Options":["DENY"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"]}}
{"level":"error","ts":1689263652.5236485,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000096315,"size":27,"status":400,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"],"Content-Security-Policy":["default-src 'self'"],"Content-Type":["text/plain; charset=utf-8"]}}
{"level":"error","ts":1689263652.52849,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000098102,"size":27,"status":400,"resp_headers":{"Content-Security-Policy":["default-src 'self'"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"]}}
{"level":"error","ts":1689263652.5350783,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"],"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000092981,"size":27,"status":400,"resp_headers":{"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"],"Content-Security-Policy":["default-src 'self'"],"Content-Type":["text/plain; charset=utf-8"]}}
{"level":"error","ts":1689263652.5437987,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263652.55607,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263652.5698211,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263652.5719001,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263661.3029912,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"],"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000124302,"size":27,"status":400,"resp_headers":{"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Xss-Protection":["1; mode=block"],"Content-Security-Policy":["default-src 'self'"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"]}}
{"level":"error","ts":1689263661.3093398,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"],"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000124899,"size":27,"status":400,"resp_headers":{"X-Xss-Protection":["1; mode=block"],"Content-Security-Policy":["default-src 'self'"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"]}}
{"level":"error","ts":1689263661.3143332,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"],"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000066336,"size":27,"status":400,"resp_headers":{"Content-Security-Policy":["default-src 'self'"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"]}}
{"level":"error","ts":1689263661.3252246,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263661.3276122,"logger":"http.log.access","msg":"handled request","request":{"remote_ip":"82.120.27.128","remote_port":"52594","proto":"HTTP/2.0","method":"GET","host":"api.pepper.fit","uri":"/.well-known/mercure","headers":{"Accept-Language":["fr,fr-FR;q=0.8,en-US;q=0.5,en;q=0.3"],"Accept":["*/*"],"User-Agent":["Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:104.0) Gecko/20100101 Firefox/104.0"],"Caddy-Push":["1"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"api.pepper.fit"}},"user_id":"","duration":0.000127371,"size":27,"status":400,"resp_headers":{"Content-Security-Policy":["default-src 'self'"],"X-Frame-Options":["DENY"],"Content-Type":["text/plain; charset=utf-8"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Content-Type-Options":["nosniff"],"X-Xss-Protection":["1; mode=block"]}}
{"level":"error","ts":1689263661.3431995,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263661.3562398,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}
{"level":"error","ts":1689263661.3671107,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"http2: stream closed"}

[francislavoie]

Please enable the debug global option. The debug logs might have relevant details.

[mholt]

Is that still from beta 2? Is there anything that changed or happened that might have triggered the error to start happening?