Failed to serve quic connection

[Barre]

While running cloudflared using --protocol quic, our origin fails a few times per day with the error:

Nov 17 02:24:46  ERR Failed to serve quic connection, err: failed to accept QUIC stream: timeout: no recent network activity
Nov 17 07:01:42 ERR Failed to serve quic connection, err: failed to accept QUIC stream: timeout: no recent network activity
[...]

When it happens, cloudflared seems to register a new connection. Before a new connection is registered, cloudflare returns 502 - Bad gateway responses while accessing our services.

Nov 17 15:08:31 2021-11-17T14:08:31Z INF Connection f5a6067a-bb51-41e9-abae-539e56707072 registered connIndex=2 location=AMS

[sifrah]

The exact same thing happens to us ! Any news ?

[nmldiegues]

Can you share the cloudflared logs that evidence the problem, and share the tunnel UUID for the matching logs?

[Barre]

From our side, some more logs:

Nov 17 07:01:42 : 2021-11-17T06:01:42Z INF Connection 97a9b7a2-3e5c-4761-ab58-15e1bc1b8794 registered connIndex=1 location=FRA
Nov 17 07:01:42 : 2021-11-17T06:01:42Z ERR Failed to serve quic connection, err: failed to accept QUIC stream: timeout: no recent network activity
Nov 17 07:01:42 : 2021-11-17T06:01:42Z INF Connection 0ea59ba3-9dde-46ec-859f-669f88bd9deb registered connIndex=3 location=FRA
Nov 17 15:08:31 : 2021-11-17T14:08:31Z ERR Failed to serve quic connection, err: failed to accept QUIC stream: timeout: no recent network activity
Nov 17 15:08:31 : 2021-11-17T14:08:31Z INF Connection f5a6067a-bb51-41e9-abae-539e56707072 registered connIndex=2 location=AMS

Tunnel id: 686bc9df-d01d-4ed8-9e37-099ae8353efe

[nmldiegues]

I think you may be hitting an edge case we have identified --- which should be independent of the transport protocol used to talk to our edge --- and that's being worked on. If that's so, it'll be fixed on our end and you won't have to do anything to get it fixed.

Note that having connections dropping and reconnecting is normal and expected --- particularly if it does not happen with a rapid cadence. What's not expected is for client requests to fail, as long as 1 connection is still up between cloudflared and Cloudflare edge.

[Barre]

Perfect, thank you!

I'll continue monitoring on our end and inform you if the issue persists.

[msiherb]

Same error popped up in our logs too but its not consistent. 70 hits in the last 7 days. In fact it happened only during the following times.

[Barre]

The issue did not occur again for us today.

[adyanth]

I can see this same issue in the logs, and the behaviour is that I see a 524 error on the webpage. A sample Ray ID: 6bd982b7da924aec

[ivanjx]

in my case this happens after i sent many http requests (like seeking through an hls video feed quickly) in a short amount of time then it will give 524. if i switch my network (client device) from wifi to mobile data for example then it starts responding again and if i switched back to the same wifi then it gives me 524 again. so it seems that 524 is tied to one type of connection.

[user8446]

The errors stopped for me when switching back to http2, so might not be an edge case. Others would have to try http2 to see if this is in indeed quic related.

[buroa]

Same thing happening to me. I had to switch back to http2 because too many eyeballs are getting errors from time to time. It also seems to be a particular connection, not all clients have the same problem. Some can connect and some can't. I have tried to switch to quic many times over the past few weeks but it results in the same stalled connections. It's completely random, but I looked at my graphs and saw this weird spike on one of Cloudflared connections. There definitely wasn't an influx of requests, it just seemed something was stuck.

The tunnel/cloudflared sits inside a k8s cluster, and does scale up and down from time to time. It routes all requests to the Contour/Envoy ingress controller.

I also see these logs:

Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2
Failed to serve quic connection error="timeout: no recent network activity" connIndex=2

Tunnel ID: b98a45ec-4647-489c-b57a-93cc9051fcdd

[buroa]

Adding a bit more context here... I think this is due to when a backend issues a TCP RST (or hangs too long)... In my deployment, it is like so:

Cloudflared <-> Contour Envoy <-> Backend

When the backend takes too long to reply, Envoy breaks the backend connection (TCP RST), but Cloudflared doesn't realize it and then the entire flow breaks as well as clients cannot connect (origin timeouts). I even created multiple origin tunnels (Cloudflare Load Balancer) and still, even one connection having this issue will break all origin tunnels.

PSA: If this happens on http2, the Cloudflare Edge terminates the origin connection, which is quite annoying if you have a shell to your server, because that is destroyed as well.

[michaelwu0505]

I can confirm that cloudflared version 2022.2.0 (built 2022-02-04-1113 UTC) also has this problem. I encountered the same issue a few days ago.

[nmldiegues]

We have done some work/fixes for QUIC protocol transport in both cloudflared and our edge. Would you mind letting us know how this goes with latest cloudflared version 2022.2.2?

[ivanjx]

done some testing with hls streaming and seems to be working better with the latest version. will continue to monitor for a few days

[nmldiegues]

We'll close the issue for now. We'd appreciate a new one being open with new evidence, should it happen, so we do not conflate a lot of confusing evidence in the same thread over a long period of time.

[achilles-git]

@nmldiegues I think you may be hitting an edge case we have identified is it possible to tell what was the edge case here ??

We have done some work/fixes for QUIC protocol transport in both cloudflared and our edge. and what was the fix done at these sides ??

[kienzlenator]

ERR Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0

This continues to be an issue. I am using the Cloudflared addon with Home Assistant and it is next to impossible to keep my tunnel on line. For the record, I buy my domain directly from Cloudflare so that probably isn't the issue here.

[Rijndael1998]

2022-11-22T18:55:12Z INF Starting tunnel tunnelID=83e89690-936f-4d28-a2d0-8abcc4473e71
2022-11-22T18:55:12Z INF Version 2022.10.3
2022-11-22T18:55:12Z INF GOOS: linux, GOVersion: go1.18.6, GoArch: amd64
2022-11-22T18:55:12Z INF Settings: map[config:.cloudflared/config.yml cred-file:/home/x/.cloudflared/83e89690-936f-4d28-a2d0-8abcc4473e71.json credentials-file:/home/x/.cloudflared/83e89690-936f-4d28-a2d0-8abcc4473e71.json url:http://127.0.0.1:80]
2022-11-22T18:55:12Z INF cloudflared will not automatically update if installed by a package manager.
2022-11-22T18:55:12Z INF Generated Connector ID: 1453508b-213a-4109-8116-a8d2d4f9dfb1
2022-11-22T18:55:12Z INF Initial protocol quic
2022-11-22T18:55:12Z INF ICMP proxy will use 192.168.0.x as source for IPv4
2022-11-22T18:55:12Z INF ICMP proxy will use fe80::96de:x:x:x in zone enp2s0 as source for IPv6
2022-11-22T18:55:12Z INF Starting metrics server on 127.0.0.1:46867/metrics
2022/11/22 18:55:12 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022-11-22T18:55:12Z WRN Your version 2022.10.3 is outdated. We recommend upgrading it to 2022.11.0
2022-11-22T18:55:22Z ERR Failed to serve quic connection error="Application error 0x0" connIndex=0 ip=198.41.x.x
2022-11-22T18:55:22Z ERR Serve tunnel error error="Application error 0x0" connIndex=0 ip=198.41.x.x
2022-11-22T18:55:22Z INF Retrying connection in up to 2s connIndex=0 ip=198.41.x.x
2022-11-22T18:55:24Z INF Tunnel server stopped
2022-11-22T18:55:24Z ERR Initiating shutdown error="Application error 0x0"
2022-11-22T18:55:24Z ERR icmp router terminated error="context canceled"
2022-11-22T18:55:24Z INF Metrics server stopped
Application error 0x0

I'm having this issue rn. I literally changed nothing.

[CrisElGeek]

Today i had the same problem, i just changed my default dns IP addresses to the OpenDNS ones and it solved the problem (you could use others like google's or the cloudflare ones)

[Rijndael1998]

The problem resolved itself. I'm incredibly confused.

[otherguy]

I have the same exact problem that @Rijndael1998 mentioned since today. No changes on our side.

@nmldiegues perhaps this could be reopened?

[blablazzz]

me too. Everything was working, and yesterday i got same errors: WRN Failed to serve quic connection error="timeout: no recent network activity"

[timscott]

I'm getting this consistently. When I hit the site I get 1033 Argo tunnel error

[otherguy]

@nmldiegues we also still see this happening. Could you please reopen?

[sudarshan-reddy]

Like Nuno says in the earlier comment, can you open a new issue with the details?

[waptik]

I'm also seeing this error in my Connector diagnostics's live logs. When i use tunnels with custom domain proxied to my deno fresh app, the app auto refreshes every second. But that doesn't seem to be the case for quick tunnel proxies. I wonder why

[Dutchy-79]

Also got this error Using a Cloudflare domain and the home assistant addon. It started after a power loss and now several reboots and fresh installations still the same. Also in zero trust the tunnel goes to down after i create a new one.

Tunnel-ID 52a82c6e-cb73-4d72-b1a5-94ab0fb29428

[Dutchy-79]

---

Add-on: Cloudflared Use a Cloudflare Tunnel to remotely connect to Home Assistant without opening any ports

Add-on version: 4.1.9 You are running the latest version of this add-on. System: Home Assistant OS 10.3 (amd64 / qemux86-64) Home Assistant Core: 2023.6.3 Home Assistant Supervisor: 2023.06.2

Please, share the above information when looking for help or support in, e.g., GitHub, forums or the Discord chat.

[13:26:10] INFO: Checking add-on config... [13:26:11] INFO: Checking for existing certificate... [13:26:11] INFO: Existing certificate found [13:26:11] INFO: Checking for existing tunnel... [13:26:11] INFO: Existing tunnel with ID 52a82c6e-cb73-4d72-b1a5-94ab0fb29428 found [13:26:11] INFO: Checking if existing tunnel matches name given in config [13:26:12] INFO: Existing Cloudflare Tunnel name matches config, proceeding with existing tunnel file [13:26:12] INFO: Creating config file... [13:26:22] INFO: Validating config file... Validating rules from /tmp/config.json OK [13:26:23] INFO: Creating DNS entry MYDOMAIN... 2023-06-25T11:26:25Z INF Added CNAME MYDOMAIN which will route to this tunnel tunnelID=52a82c6e-cb73-4d72-b1a5-94ab0fb29428 [13:26:26] INFO: Finished setting up the Cloudflare Tunnel [13:26:33] INFO: Connecting Cloudflare Tunnel... 2023-06-25T11:26:34Z INF Starting tunnel tunnelID=52a82c6e-cb73-4d72-b1a5-94ab0fb29428 2023-06-25T11:26:34Z INF Version 2023.6.1 2023-06-25T11:26:34Z INF GOOS: linux, GOVersion: go1.19.6, GoArch: amd64 2023-06-25T11:26:34Z INF Settings: map[config:/tmp/config.json cred-file:/data/tunnel.json credentials-file:/data/tunnel.json loglevel:info metrics:0.0.0.0:36500 no-autoupdate:true origincert:/data/cert.pem] 2023-06-25T11:26:34Z INF Generated Connector ID: 663b0fec-bcd0-43fd-8a56-442923dbf5d9 2023-06-25T11:26:34Z INF Initial protocol quic 2023-06-25T11:26:34Z INF ICMP proxy will use 172.30.33.7 as source for IPv4 2023-06-25T11:26:34Z INF ICMP proxy will use :: as source for IPv6 2023-06-25T11:26:34Z INF Starting metrics server on [::]:36500/metrics 2023-06-25T11:26:34Z INF Registered tunnel connection connIndex=0 connection=90107efd-026d-446d-9bce-6464681e9e0e event=0 ip=198.41.192.77 location=AMS protocol=quic 2023-06-25T11:26:34Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"MYDOMAIN\",\"originRequest\":{\"noTLSVerify\":true},\"service\":\"http://MYIP:PORT\"},{\"originRequest\":{\"noTLSVerify\":true},\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=2 2023-06-25T11:26:35Z INF Registered tunnel connection connIndex=1 connection=864d5850-96dc-4505-8f20-01a4ff7cdeec event=0 ip=198.41.200.193 location=BRU protocol=quic 2023-06-25T11:26:36Z INF Registered tunnel connection connIndex=2 connection=9ec12e6f-4c98-4d99-8e46-a686d82a09fa event=0 ip=198.41.200.63 location=BRU protocol=quic 2023-06-25T11:26:37Z INF Registered tunnel connection connIndex=3 connection=5f49849d-f8ae-45c5-8fad-5cdf436e06be event=0 ip=198.41.192.107 location=AMS protocol=quic 2023-06-25T11:26:57Z INF Unregistered tunnel connection connIndex=0 event=0 ip=198.41.192.77 2023-06-25T11:26:57Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.77 2023-06-25T11:26:57Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.77 2023-06-25T11:26:57Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.192.77 2023-06-25T11:26:57Z INF Unregistered tunnel connection connIndex=3 event=0 ip=198.41.192.107 2023-06-25T11:26:57Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.107 2023-06-25T11:26:57Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.107 2023-06-25T11:26:57Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.192.107 2023-06-25T11:26:57Z INF Unregistered tunnel connection connIndex=1 event=0 ip=198.41.200.193 2023-06-25T11:26:57Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.63 2023-06-25T11:26:57Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.200.193 2023-06-25T11:26:57Z ERR Serve tunnel error error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.200.193 2023-06-25T11:26:57Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.200.193 2023-06-25T11:26:57Z ERR Failed to serve quic connection error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.63 2023-06-25T11:26:57Z ERR Serve tunnel error error="timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.63 2023-06-25T11:26:57Z INF Retrying connection in up to 1s connIndex=2 event=0 ip=198.41.200.63 2023-06-25T11:26:58Z ERR Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 2023-06-25T11:26:58Z ERR Connection terminated error="timeout: no recent network activity" connIndex=1 2023-06-25T11:26:59Z ERR Connection terminated error="timeout: no recent network activity" connIndex=2 2023-06-25T11:26:59Z ERR Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 2023-06-25T11:27:10Z INF Registered tunnel connection connIndex=0 connection=97cb7660-f74a-455b-a752-cc880e42c2ea event=0 ip=198.41.192.77 location=AMS protocol=quic 2023-06-25T11:27:10Z INF Registered tunnel connection connIndex=3 connection=5e72523c-8160-4367-85ab-0cc1cb7ad843 event=0 ip=198.41.192.107 location=AMS protocol=quic 2023-06-25T11:27:10Z INF Registered tunnel connection connIndex=2 connection=ba0bb32f-83bb-43a9-aa04-dfe13a4f6bdf event=0 ip=198.41.200.43 location=BRU protocol=quic 2023-06-25T11:27:10Z INF Registered tunnel connection connIndex=1 connection=b2b29277-9794-433d-8982-451eaa1b490f event=0 ip=198.41.200.53 location=BRU protocol=quic

[hahahaha123567]

exec reset if you just change your time zone config

[mystiquewolf]

+1 for this issue

[hillz2]

+1 same issue here

[cheonmux]

+1 same issue here

{"level":"error","event":0,"ip":"198.41.200.13","connIndex":3,"error":"context canceled","time":"2023-09-06T07:49:03Z","message":"Failed to serve quic connection"}
{"level":"error","event":0,"ip":"198.41.200.233","connIndex":1,"error":"context canceled","time":"2023-09-06T07:49:03Z","message":"Failed to serve quic connection"}
{"level":"error","event":0,"ip":"198.41.192.37","connIndex":0,"error":"context canceled","time":"2023-09-06T07:49:04Z","message":"Failed to serve quic connection"}
{"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"context canceled","time":"2023-09-06T07:49:04Z","message":"Failed to serve quic connection"}

[kingname]

Same issue today

[jrencardoso]

+1. Same issue observed when connecting Immich (iOS app) with the server.

[matti]

not sure if related but in docker container suddenly cloudflared says Serve tunnel error error="failed to accept QUIC stream: Application error 0x0 (remote)

[GlitchWitch]

not sure if related but in docker container suddenly cloudflared says Serve tunnel error error="failed to accept QUIC stream: Application error 0x0 (remote)

We're seeing this too. Though it's potentially related to this: https://www.cloudflarestatus.com/incidents/s1hkh315y9s9

Edit: Seems to be working again as of resolution of this incident.

[m0x61h0x64i]

same, error while using cloudflared cli tool for using Zero Trust:

ERR Serve tunnel error error="Application error 0x0 (remote)"

[matti]

Yep, it was that outage. Also found out that cloudflared just gives up and exits :oOn Oct 25, 2023, at 20:03, m0x61h0x64i @.***> wrote: same, error while using cloudflared cli tool for using Zero Trust: ERR Serve tunnel error error="Application error 0x0 (remote)"

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[MrZah]

I am still experiencing this problem:

2023-10-27T06:06:48Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.227 2023-10-27T06:06:48Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.192.227 2023-10-27T06:06:48Z INF Unregistered tunnel connection connIndex=1 event=0 ip=198.41.192.67 2023-10-27T06:06:48Z ERR Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.67 2023-10-27T06:06:48Z ERR Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1 event=0 ip=198.41.192.67 2023-10-27T06:06:48Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.192.67 2023-10-27T06:06:48Z ERR Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1 2023-10-27T06:06:48Z ERR Connection terminated error="timeout: no recent network activity" connIndex=2 2023-10-27T06:06:49Z ERR Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=0 2023-10-27T06:06:49Z ERR Connection terminated error="timeout: no recent network activity" connIndex=3 2023-10-27T06:07:00Z INF Registered tunnel connection connIndex=0 connection=055349d6-ef0c-4710-94b2-91813de6f2b0 event=0 ip=198.41.200.33 location=cph01 protocol=quic 2023-10-27T06:07:00Z INF Registered tunnel connection connIndex=1 connection=6f2668d9-eaf3-42fe-b368-626c5143bd36 event=0 ip=198.41.192.67 location=fra06 protocol=quic 2023-10-27T06:07:01Z INF Registered tunnel connection connIndex=2 connection=7b62041a-f76b-4688-85c9-2313dab89e68 event=0 ip=198.41.200.43 location=cph01 protocol=quic 2023-10-27T06:07:01Z INF Registered tunnel connection connIndex=3 connection=26af7434-dcf1-43ff-8192-b52683e94b22 event=0 ip=198.41.192.7 location=fra12 protocol=quic

[ometecuhtli2001]

Also experiencing this problem 100% of the time for at least the last couple of weeks. No errors, just connection attempts time out. Running two cloudflared v2023.8.2 instances on VMs running Oracle Linux 8.8. I’ve got e-mail alerts configured and when I dug deeper into this problem and tried a cloudflared cleanup I got alerts saying the tunnel was down. I recycled the service on each VM, got an e-mail saying tunnel status was healthy, but I still cannot connect.

root@cloudflared2 cloudflared]# tail -f cloudflared.log {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:24Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:24Z","message":"Serve tunnel error"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:25Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:25Z","message":"Serve tunnel error"} {"level":"warn","error":"timeout: no recent network activity","connIndex":2,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"warn","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":3,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"warn","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":0,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:27Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:27Z","message":"Serve tunnel error"} {"level":"error","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":1,"time":"2023-10-27T14:06:27Z","message":"Connection terminated"} {"level":"error","time":"2023-10-28T02:57:42Z","message":"cloudflared was installed by a package manager. Please update using the same method."} {"level":"warn","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:18Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:18Z","message":"Serve tunnel error"} {"level":"warn","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":1,"time":"2023-10-28T02:59:19Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:25Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:25Z","message":"Serve tunnel error"} {"level":"warn","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":3,"time":"2023-10-28T02:59:27Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T02:59:31Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T02:59:31Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":0,"time":"2023-10-28T02:59:33Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:34Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:34Z","message":"Serve tunnel error"} {"level":"error","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":2,"time":"2023-10-28T02:59:35Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.200.53","connIndex":0,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.200.43","connIndex":3,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","time":"2023-10-28T03:09:48Z","message":"no more connections active and exiting"} {"level":"error","error":"context canceled","time":"2023-10-28T03:09:48Z","message":"icmp router terminated"} {"level":"error","time":"2023-10-28T03:15:23Z","message":"writing release: Application error 0x0 (remote)"} {"level":"warn","event":0,"ip":"198.41.192.27","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.27","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Serve tunnel error"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":1,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":1,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":1,"time":"2023-10-28T03:15:24Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.33","connIndex":3,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:28Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.33","connIndex":3,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:28Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":3,"time":"2023-10-28T03:15:30Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:32Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:32Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":2,"time":"2023-10-28T03:15:33Z","message":"Connection terminated"}

[matti]

When this happens, does cloudflared also exit?On Oct 28, 2023, at 06:29, Steve Miller @.> wrote: Also experiencing this problem 100% of the time for at least the last couple of weeks. No errors, just connection attempts time out. Running two cloudflared v2023.8.2 instances on VMs running Oracle Linux 8.8. I’ve got e-mail alerts configured and when I dug deeper into this problem and tried a cloudflared cleanup I got alerts saying the tunnel was down. I recycled the service on each VM, got an e-mail saying tunnel status was healthy, but I still cannot connect. @. cloudflared]# tail -f cloudflared.log {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:24Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:24Z","message":"Serve tunnel error"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:25Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:25Z","message":"Serve tunnel error"} {"level":"warn","error":"timeout: no recent network activity","connIndex":2,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"warn","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":3,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"warn","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":0,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:27Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:27Z","message":"Serve tunnel error"} {"level":"error","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":1,"time":"2023-10-27T14:06:27Z","message":"Connection terminated"} {"level":"error","time":"2023-10-28T02:57:42Z","message":"cloudflared was installed by a package manager. Please update using the same method."} {"level":"warn","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:18Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:18Z","message":"Serve tunnel error"} {"level":"warn","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":1,"time":"2023-10-28T02:59:19Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:25Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:25Z","message":"Serve tunnel error"} {"level":"warn","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":3,"time":"2023-10-28T02:59:27Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T02:59:31Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T02:59:31Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":0,"time":"2023-10-28T02:59:33Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:34Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:34Z","message":"Serve tunnel error"} {"level":"error","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":2,"time":"2023-10-28T02:59:35Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.200.53","connIndex":0,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.200.43","connIndex":3,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","time":"2023-10-28T03:09:48Z","message":"no more connections active and exiting"} {"level":"error","error":"context canceled","time":"2023-10-28T03:09:48Z","message":"icmp router terminated"} {"level":"error","time":"2023-10-28T03:15:23Z","message":"writing release: Application error 0x0 (remote)"} {"level":"warn","event":0,"ip":"198.41.192.27","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.27","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Serve tunnel error"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":1,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":1,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":1,"time":"2023-10-28T03:15:24Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.33","connIndex":3,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:28Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.33","connIndex":3,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:28Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":3,"time":"2023-10-28T03:15:30Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:32Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:32Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":2,"time":"2023-10-28T03:15:33Z","message":"Connection terminated"}

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

[ometecuhtli2001]

The cloudflared process on each VM continues to run.

On Fri, Oct 27, 2023 at 22:21 Matti Paksula @.***> wrote:

When this happens, does cloudflared also exit?On Oct 28, 2023, at 06:29, Steve Miller @.> wrote: Also experiencing this problem 100% of the time for at least the last couple of weeks. No errors, just connection attempts time out. Running two cloudflared v2023.8.2 instances on VMs running Oracle Linux 8.8. I’ve got e-mail alerts configured and when I dug deeper into this problem and tried a cloudflared cleanup I got alerts saying the tunnel was down. I recycled the service on each VM, got an e-mail saying tunnel status was healthy, but I still cannot connect. @. cloudflared]# tail -f cloudflared.log {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:24Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:24Z","message":"Serve tunnel error"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:25Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:25Z","message":"Serve tunnel error"} {"level":"warn","error":"timeout: no recent network activity","connIndex":2,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"warn","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":3,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"warn","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":0,"time":"2023-10-27T14:06:26Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:27Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: timeout: no recent network activity","time":"2023-10-27T14:06:27Z","message":"Serve tunnel error"} {"level":"error","error":"failed to accept QUIC stream: timeout: no recent network activity","connIndex":1,"time":"2023-10-27T14:06:27Z","message":"Connection terminated"} {"level":"error","time":"2023-10-28T02:57:42Z","message":"cloudflared was installed by a package manager. Please update using the same method."} {"level":"warn","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:18Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.37","connIndex":1,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:18Z","message":"Serve tunnel error"} {"level":"warn","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":1,"time":"2023-10-28T02:59:19Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:25Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":3,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:25Z","message":"Serve tunnel error"} {"level":"warn","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":3,"time":"2023-10-28T02:59:27Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T02:59:31Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.53","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T02:59:31Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":0,"time":"2023-10-28T02:59:33Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:34Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"failed to accept QUIC stream: Application error 0x0 (remote)","time":"2023-10-28T02:59:34Z","message":"Serve tunnel error"} {"level":"error","error":"failed to accept QUIC stream: Application error 0x0 (remote)","connIndex":2,"time":"2023-10-28T02:59:35Z","message":"Connection terminated"} {"level":"error","event":0,"ip":"198.41.200.53","connIndex":0,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.37","connIndex":1,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.192.47","connIndex":2,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","event":0,"ip":"198.41.200.43","connIndex":3,"error":"context canceled","time":"2023-10-28T03:09:48Z","message":"Failed to serve quic connection"} {"level":"error","time":"2023-10-28T03:09:48Z","message":"no more connections active and exiting"} {"level":"error","error":"context canceled","time":"2023-10-28T03:09:48Z","message":"icmp router terminated"} {"level":"error","time":"2023-10-28T03:15:23Z","message":"writing release: Application error 0x0 (remote)"} {"level":"warn","event":0,"ip":"198.41.192.27","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.27","connIndex":0,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Serve tunnel error"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":1,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.43","connIndex":1,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:23Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":1,"time":"2023-10-28T03:15:24Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.200.33","connIndex":3,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:28Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.200.33","connIndex":3,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:28Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":3,"time":"2023-10-28T03:15:30Z","message":"Connection terminated"} {"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:32Z","message":"Failed to serve quic connection"} {"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"Application error 0x0 (remote)","time":"2023-10-28T03:15:32Z","message":"Serve tunnel error"} {"level":"warn","error":"Application error 0x0 (remote)","connIndex":2,"time":"2023-10-28T03:15:33Z","message":"Connection terminated"}

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you commented.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/cloudflare/cloudflared/issues/510#issuecomment-1783706690, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGL6U2JXTVIUOVRHER7XHZLYBSI45AVCNFSM5IHDBT7KU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZYGM3TANRWHEYA . You are receiving this because you commented.Message ID: @.***>

[DmytroRybka]

Same issue 2023-11-14T04:39:53Z ERR Failed to serve quic connection error="timeout: no recent network activity" connIndex=0 event=0

[filipduch]

Same issue,

2023-12-07T08:47:34Z ERR Failed to serve quic connection error="timeout: no recent network activity"

The worst part is that the requests are dropped even though the LoadBalancer should retry the requests to a different origin but for some reason, it does not (Zero-downtime-failover feature). Does anyone know why?

[schlumm]

Also here. Has anyone solved it?

2024-01-01T19:47:02Z INF Starting tunnel tunnelID=88f65771-27ef-4cee-af11-6b25ce4400ee 2024-01-01T19:47:02Z INF Version 2023.10.0 2024-01-01T19:47:02Z INF GOOS: linux, GOVersion: go1.20.6, GoArch: arm64

[Nenodema]

Still issues on our site..

Connection terminated error="failed to accept QUIC stream

[germanikus666]

I have the same error. Is there already a solution for this?

[schlumm]

I have changed the domain to xyz instead of cf. After applying all the changes, it started to work. For me, the wifi at my work probably dont allow using the xyz domain, so I need to switch to mobile data.

[fantnhu]

2024-03-24T15:30:20Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=1 event=0 ip=198.41.200.xx
2024-03-24T15:30:20Z INF Retrying connection in up to 1s connIndex=1 event=0 ip=198.41.200.xx
2024-03-24T15:30:20Z INF Unregistered tunnel connection connIndex=3 event=0 ip=198.41.192.xx
2024-03-24T15:30:20Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.xx
2024-03-24T15:30:20Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=3 event=0 ip=198.41.192.xx
2024-03-24T15:30:20Z INF Retrying connection in up to 1s connIndex=3 event=0 ip=198.41.192.xx
2024-03-24T15:30:21Z WRN Connection terminated error="timeout: no recent network activity" connIndex=3
2024-03-24T15:30:22Z WRN Connection terminated error="timeout: no recent network activity" connIndex=1
2024-03-24T15:30:26Z INF Unregistered tunnel connection connIndex=2 event=0 ip=198.41.200.xx
2024-03-24T15:30:26Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2 event=0 ip=198.41.200.xx

 Add-on version: 5.1.7
 You are running the latest version of this add-on.
 System: Home Assistant OS 12.1  (amd64 / generic-x86-64)
 Home Assistant Core: 2024.3.3
 Home Assistant Supervisor: 2024.03.1