Open prvashisht opened 2 years ago
sudarshan-reddy
commented
2 years ago Hey @prvashisht ! Thanks for the detailed report.
Looking through your logs it does seem to me like cloudflared is reconnecting even if some of these connections were dropped. Is your internet connection stable? Are you losing connectivity to your tunnel?
prvashisht
commented
2 years ago @sudarshan-reddy yes, the connection is stable, I have 1Gbit connection and speedtest shows it's been working fine (at least in the last two days)
Is there a way (and is it possible) to get more detailed logs? It's because I'm seeing this problem regardless of my installation method.
sudarshan-reddy
commented
2 years ago Connection drops are expected (thats why we have 4 and reconnect). However, the periodicity of your connection drop does seem rather high. Can you temporarily add protocol: http2 to your config.yaml and let me know what happens?
prvashisht
commented
2 years ago Now I'm having a different problem, I was getting this page on every subdomain I configured.
So I migrated the tunnel from locally configured to remotely managed one, the same Unable to connect to origin problem persists. I'm really new to tinkering with my network and for now I feel lost. I just tried tailscale and I'm going to use that and will drop trying to make this work.
I couldn't make http2 protocol work long enough to get any useful logs.
don4of4
commented
2 years ago Hey @sudarshan-reddy we are seeing similar (aggressive) connection drops going out of azure southcentraus AKS on quic. More eyeballs on these connection faults would be helpful. See #2535568.
iAmSaugata
commented
2 years ago @sudarshan-reddy yes, the connection is stable, I have 1Gbit connection and speedtest shows it's been working fine (at least in the last two days)
Is there a way (and is it possible) to get more detailed logs? It's because I'm seeing this problem regardless of my installation method.
Is this graph representing correctly? It looks like your upload speed it very low, and your CF tunnel is actually dependent on incoming speed and stability, and as per the graph, it looks like very unstable.
I assume, in the graph, every block representing 30 minutes, and if you looks closely, you will see there are multiple drop in download and upload speed, it could be possible that the actual connection getting dropped in between those 30 minutes. Please validated your CF logs with those timings.
prvashisht
commented
2 years ago @iAmSaugata I have stopped using Cloudflare for now, and I will in the near future look for a more granulated graph timed with CF logs. However, the speed shown is correct. My internet connection has a 1Gibps download and 50Mbps upload speed. Some drops to ~150Mbps are expected in the DL speed.
Kiran01bm
commented
2 years ago So we are hitting the same issue and the background is that we are migrating from a dynamic tunnel to named one's. The daemon version 2022.8.4.
2022-09-08 12:35:212022/09/08 02:35:21 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022/09/08 02:35:21 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
nmldiegues
commented
2 years ago So we are hitting the same issue and the background is that we are migrating from a dynamic tunnel to named one's. The daemon version
2022.8.4.2022-09-08 12:35:212022/09/08 02:35:21 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details. 2022/09/08 02:35:21 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
That is just a warning to let you know you can improve the performance of the QUIC transport used by the tunnel by tweaking the UDP buffers allowed by the kernel
it does not compromise functionality at all; it will all still just work
morpig
commented
1 year ago Experiencing similar error messages on 2022.10.3, lost all 4 HAs within 10 seconds. Running in kubernetes setup.
2022-11-11T21:10:53Z ERR error="Incoming request ended abruptly: context canceled" cfRay=768a04610c4fa089-SIN ingressRule=2 originService=http://kong-proxy.kong.svc.cluster.local:8080
2022-11-12T00:00:22Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 ip=198.41.192.227
2022-11-12T00:00:22Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3 ip=198.41.192.227
2022-11-12T00:00:22Z INF Retrying connection in up to 1s connIndex=3 ip=198.41.192.227
2022-11-12T00:00:23Z WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=3
2022-11-12T00:00:27Z INF Unregistered tunnel connection connIndex=2
2022-11-12T00:00:27Z WRN Failed to serve quic connection error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2 ip=198.41.200.53
2022-11-12T00:00:27Z WRN Serve tunnel error error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2 ip=198.41.200.53
2022-11-12T00:00:27Z INF Retrying connection in up to 1s connIndex=2 ip=198.41.200.53
2022-11-12T00:00:28Z INF Unregistered tunnel connection connIndex=0
2022-11-12T00:00:28Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=0 ip=198.41.200.113
2022-11-12T00:00:28Z WRN Serve tunnel error error="timeout: no recent network activity" connIndex=0 ip=198.41.200.113
2022-11-12T00:00:28Z INF Retrying connection in up to 1s connIndex=0 ip=198.41.200.113
2022-11-12T00:00:28Z WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=2
2022-11-12T00:00:28Z INF Unregistered tunnel connection connIndex=1
2022-11-12T00:00:28Z WRN Failed to serve quic connection error="timeout: no recent network activity" connIndex=1 ip=198.41.192.67
2022-11-12T00:00:28Z ERR Serve tunnel error error="timeout: no recent network activity" connIndex=1 ip=198.41.192.67
2022-11-12T00:00:28Z INF Retrying connection in up to 1s connIndex=1 ip=198.41.192.67
2022-11-12T00:00:28Z INF Connection 8c637052-3a44-4061-a55e-754cf0b9bef3 registered connIndex=2 ip=198.41.200.53 location=SIN
2022-11-12T00:00:29Z INF Connection d0460cc6-9de2-45fc-8b04-4963a24a0232 registered connIndex=3 ip=198.41.192.227 location=NRT
2022-11-12T00:00:29Z WRN Connection terminated error="timeout: no recent network activity" connIndex=1
2022-11-12T00:00:30Z INF Connection 42055beb-e80f-4523-a35c-fa8613698a9f registered connIndex=0 ip=198.41.200.113 location=SIN
2022-11-12T00:00:36Z INF Connection b8279b12-2b29-4e74-a549-d73dd721d4ca registered connIndex=1 ip=198.41.192.67 location=NRT
2022-11-12T00:21:24Z ERR Failed to handle QUIC stream error=EOF connIndex=0 ip=198.41.200.113
Thankfully the other replica immediately took over the traffic.
Should I send debug files over ticket?
joliveirinha
commented
1 year ago @morpig, that should be ok.
Describe the bug I'm trying to install Cloudflare tunnel on my raspberry pi 4b. I've tried using docker images, configuring from the zero trust dashboard, and installing via CLI. The tunnel starts, and I see four registered connections, however, the connections drop within minutes, if not seconds, and the cycle repeats over the course of a few minutes. I also created an issue (https://github.com/brenner-tobias/addon-cloudflared/issues/152) with an addon I'm using for Home Assistant, but it seems the issue is coming from Cloudflare.
To Reproduce Steps to reproduce the behavior: I mainly followed these steps last, after trying out official, and other methods pasted on the internet.
If it's an issue with Cloudflare Tunnel:
ingress:
Expected behavior A clear and concise description of what you expected to happen.
Environment and versions
lscpucommand)Logs and errors