search

cloudflare / cloudflared

Cloudflare Tunnel client (formerly Argo Tunnel)
https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide
Apache License 2.0
8.4k stars 729 forks source link

🐛Docker container fails to connect out; "no recent network activity" via quic or "i/o timeout" on http2 #1007

Open vlsalsa opened 12 months ago

vlsalsa commented 12 months ago

Describe the bug I am running cloudflared via the docker container on a registered cloudflare account. Neither quic nor http2 is connecting -- but I need help understanding where the failure is to properly diagnose with the network techs.

To Reproduce Steps to reproduce the behavior:

  1. Copy cloudflared docker container from the website, paste into terminal with docker running
  2. docker run cloudflare/cloudflared:latest tunnel --no-autoupdate run --token redacted

If it's an issue with Cloudflare Tunnel:

  1. Tunnel ID : 79c75e2e-1881-4324-9548-a879f42bde8a
  2. cloudflared config: docker run cloudflare/cloudflared:latest tunnel --no-autoupdate run --token

Expected behavior I expect to be able to connect out, I tried DIG, and udp out to the ips above. I have attempted nc -vzu connections to the ports as well as digs, please see logs and errors section

Logs and errors

2023-06-28T15:08:44Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 ```
or when defaulting to http2
```59Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.77:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.77`

when attempting to troubleshoot via https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/ports-and-ips/

nc -vzu 198.41.200.13 7844
Connection to 198.41.200.13 7844 port [udp/*] succeeded!
nc -vzu 198.41.200.23 7844
Connection to 198.41.200.23 7844 port [udp/*] succeeded!
dig A region1.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> A region1.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18181
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region1.v2.argotunnel.com. IN  A

;; ANSWER SECTION:
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.27
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.167
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.67
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.107
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.7
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.77
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.57
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.47
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.227
region1.v2.argotunnel.com. 6703 IN  A   198.41.192.37

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:26:27 EDT 2023
;; MSG SIZE  rcvd: 214

and 

dig AAAA region1.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> AAAA region1.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31966
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region1.v2.argotunnel.com. IN  AAAA

;; ANSWER SECTION:
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::6
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::9
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::1
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::5
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::4
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::8
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::2
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::7
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::10
region1.v2.argotunnel.com. 16879 IN AAAA    2606:4700:a0::3

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:27:13 EDT 2023
;; MSG SIZE  rcvd: 334

and 

dig A region2.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> A region2.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 494
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region2.v2.argotunnel.com. IN  A

;; ANSWER SECTION:
region2.v2.argotunnel.com. 84937 IN A   198.41.200.43
region2.v2.argotunnel.com. 84937 IN A   198.41.200.23
region2.v2.argotunnel.com. 84937 IN A   198.41.200.13
region2.v2.argotunnel.com. 84937 IN A   198.41.200.73
region2.v2.argotunnel.com. 84937 IN A   198.41.200.233
region2.v2.argotunnel.com. 84937 IN A   198.41.200.53
region2.v2.argotunnel.com. 84937 IN A   198.41.200.33
region2.v2.argotunnel.com. 84937 IN A   198.41.200.193
region2.v2.argotunnel.com. 84937 IN A   198.41.200.113
region2.v2.argotunnel.com. 84937 IN A   198.41.200.63

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:27:42 EDT 2023
;; MSG SIZE  rcvd: 214

and

dig AAAA region2.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> AAAA region2.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 350
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region2.v2.argotunnel.com. IN  AAAA

;; ANSWER SECTION:
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::9
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::10
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::7
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::8
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::2
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::1
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::3
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::6
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::5
region2.v2.argotunnel.com. 84908 IN AAAA    2606:4700:a8::4

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:28:11 EDT 2023
;; MSG SIZE  rcvd: 334

**Environment and versions**
 - OS: Ubuntu
 - Architecture: Intel Arm docker run cloudflare/cloudflared --version
cloudflared version 2023.6.1 (built 2023-06-20-0923 UTC)
 - Version: [e.g. 2022.02.0]

Additional context I wonder if it's the docker container? Unfortunately, as this machine is managed, we can not run sudo for new code installs without a thorough and lengthy review.

vlsalsa commented 12 months ago

I checked to see today if I was able to connect out using busybox from inside docker, and I was able to.

See: 

(base) user@computer~$  docker run busybox nc -vzu [198.41.192.77](http://198.41.192.77/) 7844                                      
Unable to find image 'busybox:latest' locally                                                                      
latest: Pulling from library/busybox                                                                               
809d8e20e203: Pull complete 
Digest: sha256:2376a0c12759aa1214ba83e771ff252c7b1663216b192fbe5e0fb364e952f85c                                    
Status: Downloaded newer image for busybox:latest                                                                  
[198.41.192.77](http://198.41.192.77/) 
([198.41.192.77:7844](http://198.41.192.77:7844/)) open                                                                            
(base) user@computer~$ docker run --network host busybox nc -vzu [198.41.192.77](http://198.41.192.77/) 7844
[198.41.192.77](http://198.41.192.77/) ([198.41.192.77:7844](http://198.41.192.77:7844/)) open                                                                            
(base) user@computer@computer~$ docker run --network none busybox nc -vzu [198.41.192.77](http://198.41.192.77/) 7844                      
nc: [198.41.192.77](http://198.41.192.77/) ([198.41.192.77:7844](http://198.41.192.77:7844/)): Network is unreachable
Kaijun commented 12 months ago

it seems quic never works properly

it's not docker related issue. running cloudflared service with quic protocol on host os also gets the same issue

vlsalsa commented 12 months ago

Attempted to download the .deb locally, and had the same issues outside of docker.

Running: $ wget https://github.com/cloudflare/cloudflared/releases/download/2023.6.1/cloudflared-linux-amd64 $ chmod u+x cloudflared-linux-amd64 ./cloudflared-linux-amd64 tunnel run --token

Log Output

2023-06-29T13:54:21Z INF Starting tunnel tunnelID=79c75e2e-1881-4324-9548-a879f42bde8a
2023-06-29T13:54:21Z INF Version 2023.6.1
2023-06-29T13:54:21Z INF GOOS: linux, GOVersion: go1.19.6, GoArch: amd64
2023-06-29T13:54:21Z INF Settings: map[token:*****]
2023-06-29T13:54:21Z INF Generated Connector ID: 3de1e423-b499-448d-*redacted
2023-06-29T13:54:21Z INF cloudflared will not automatically update when run from the shell. To enable auto-updates, run cloudflared as a service: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/run-tunnel/as-a-service/
2023-06-29T13:54:26Z INF Initial protocol quic
2023-06-29T13:54:26Z INF ICMP proxy will use **** as source for IPv4
2023-06-29T13:54:26Z INF ICMP proxy will use **** in zone eno0 as source for IPv6
2023-06-29T13:54:41Z INF Starting metrics server on 127.0.0.1:34429/metrics
2023-06-29T13:55:01Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.67
2023-06-29T13:55:01Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.67
2023-06-29T13:55:06Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113

Errors: 


2023-06-29T13:54:41Z INF Starting metrics server on 127.0.0.1:34429/metrics
2023-06-29T13:55:01Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.67
2023-06-29T13:55:01Z INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.67
2023-06-29T13:55:06Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.113
2023-06-29T13:55:06Z INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.200.113
2023-06-29T13:55:12Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.7
2023-06-29T13:55:12Z INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.7
2023-06-29T13:55:23Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.227
2023-06-29T13:57:18Z WRN If this log occurs persistently, and cloudflared is unable to connect to Cloudflare Network with `quic` protocol, then most likely your machine/network is getting its egress UDP to port 7844 (or others) blocked or dropped. Make sure to allow egress connectivity as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ports-and-ips/
If you are using private routing to this Tunnel, then UDP (and Private DNS Resolution) will not work unless your cloudflared can connect with Cloudflare Network with `quic`. connIndex=0 event=0 ip=198.41.200.43
2023-06-29T13:57:18Z INF Switching to fallback protocol http2 connIndex=0 event=0 ip=198.41.200.43
2023-06-29T13:57:33Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.200.233:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.233
2023-06-29T13:57:33Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.200.233:7844: i/o timeout" connIndex=0 event=0 ip=198.41.200.233
2023-06-29T13:57:33Z INF Retrying connection in up to 1s connIndex=0 event=0 ip=198.41.200.233
2023-06-29T13:57:50Z ERR Unable to establish connection with Cloudflare edge error="DialContext error: dial tcp 198.41.192.167:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.167
Kaijun commented 12 months ago

@vlsalsa did you have proxy running which is probably not able to proxy UDP traffic?

if you have, try run cloudflared tunnel without proxy

hostinger-free commented 11 months ago

This happened to me 3 days ago (1 July 2023) when I was creating a docker-compose with a cloudfared agent (image) along with some other services (as I have a handful already running like that). The new had these errors, and now an older set of services with cloudflared images that was still working yesterday, suddenly today doesn't work. Here are the typical error messages: 

ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.192.67
ERR Connection terminated error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=3
WRN Connection terminated error="failed to accept QUIC stream: timeout: no recent network activity" connIndex=1
rothn commented 11 months ago

I'm having issues too. I get errors like this and then the hosted website just gives 502 errors perpetually until cloudflared is restarted. Very annoying. For now, I probably need a script to restart it every 3.5 hours or so.

Kaijun commented 11 months ago

I'm having issues too. I get errors like this and then the hosted website just gives 502 errors perpetually until cloudflared is restarted. Very annoying. For now, I probably need a script to restart it every 3.5 hours or so.

me too, i run a nginx server exposed by cloudflare tunnel, and started a cron job to check if the nginx is accessable through cloudflare tunnel. if it fails, restart cloudflared.

rothn commented 11 months ago

I moved off this for less-sensitive stuff due to the issue FWIW. Might try this again if it is fixed. My service was sketchy and it took me quite some time to figure out the problem was actually the tunnel.

bekaertruben commented 11 months ago

I have the same issues on Fedora Server
Both on quic and http2
It started about the same as the creation of this issue, but I've not had success with any previous versions of cloudflared either 

{"level":"warn","event":0,"ip":"198.41.192.7","connIndex":2,"error":"timeout: no recent network activity","time":"2023-08-01T00:21:30Z","message":"Serve tunnel error"}
{"level":"warn","event":0,"ip":"198.41.200.113","connIndex":3,"error":"timeout: no recent network activity","time":"2023-08-01T00:21:30Z","message":"Failed to serve quic connection"}
{"level":"error","time":"2023-08-01T00:21:30Z","message":"writing call: timeout: no recent network activity"}
{"level":"warn","event":0,"ip":"198.41.200.113","connIndex":3,"error":"timeout: no recent network activity","time":"2023-08-01T00:21:30Z","message":"Serve tunnel error"}
{"level":"error","event":0,"ip":"198.41.200.53","connIndex":0,"error":"timeout: no recent network activity","time":"2023-08-01T00:21:30Z","message":"Failed to serve quic connection"}
{"level":"error","event":0,"ip":"198.41.200.53","connIndex":0,"error":"timeout: no recent network activity","time":"2023-08-01T00:21:30Z","message":"Serve tunnel error"}
{"level":"error","error":"timeout: no recent network activity","connIndex":0,"time":"2023-08-01T00:21:31Z","message":"Connection terminated"}
{"level":"error","error":"timeout: no recent network activity","connIndex":2,"time":"2023-08-01T00:21:31Z","message":"Connection terminated"}
{"level":"error","error":"timeout: no recent network activity","connIndex":3,"time":"2023-08-01T00:21:31Z","message":"Connection terminated"}
{"level":"error","error":"timeout: no recent network activity","connIndex":1,"time":"2023-08-01T00:21:32Z","message":"Connection terminated"}
brpaz commented 9 months ago

I have the same issue with with cloudflared 2023.7.3-amd64, on a k3s cluster.

I can connect to the quic port using telnet or nc from inside the network just fine, but when running cloudflared, the container enters in CrashLoop because of this error.

RR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activit │
│ INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.167                                            │
│ ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activit │
│ INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.27                                             │
│ ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activit │
│ INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.77
warjiang commented 9 months ago

Describe the bug I am running cloudflared via the docker container on a registered cloudflare account. Neither quic nor http2 is connecting -- but I need help understanding where the failure is to properly diagnose with the network techs.

To Reproduce Steps to reproduce the behavior:

  1. Copy cloudflared docker container from the website, paste into terminal with docker running
  2. docker run cloudflare/cloudflared:latest tunnel --no-autoupdate run --token redacted

If it's an issue with Cloudflare Tunnel: 4. Tunnel ID : 79c75e2e-1881-4324-9548-a879f42bde8a 5. cloudflared config: docker run cloudflare/cloudflared:latest tunnel --no-autoupdate run --token

Expected behavior I expect to be able to connect out, I tried DIG, and udp out to the ips above. I have attempted nc -vzu connections to the ports as well as digs, please see logs and errors section

Logs and errors

2023-06-28T15:08:44Z ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activity" connIndex=0 event=0 ip=198.41.200.23 ```
or when defaulting to http2
```59Z ERR Serve tunnel error error="DialContext error: dial tcp 198.41.192.77:7844: i/o timeout" connIndex=0 event=0 ip=198.41.192.77`

when attempting to troubleshoot via https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/ports-and-ips/

nc -vzu 198.41.200.13 7844
Connection to 198.41.200.13 7844 port [udp/*] succeeded!
nc -vzu 198.41.200.23 7844
Connection to 198.41.200.23 7844 port [udp/*] succeeded!
dig A region1.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> A region1.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18181
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region1.v2.argotunnel.com.   IN  A

;; ANSWER SECTION:
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.27
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.167
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.67
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.107
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.7
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.77
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.57
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.47
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.227
region1.v2.argotunnel.com. 6703   IN  A   198.41.192.37

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:26:27 EDT 2023
;; MSG SIZE  rcvd: 214

and

dig AAAA region1.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> AAAA region1.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31966
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region1.v2.argotunnel.com.   IN  AAAA

;; ANSWER SECTION:
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::6
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::9
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::1
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::5
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::4
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::8
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::2
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::7
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::10
region1.v2.argotunnel.com. 16879 IN   AAAA    2606:4700:a0::3

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:27:13 EDT 2023
;; MSG SIZE  rcvd: 334

and

dig A region2.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> A region2.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 494
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region2.v2.argotunnel.com.   IN  A

;; ANSWER SECTION:
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.43
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.23
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.13
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.73
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.233
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.53
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.33
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.193
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.113
region2.v2.argotunnel.com. 84937 IN   A   198.41.200.63

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:27:42 EDT 2023
;; MSG SIZE  rcvd: 214

and

dig AAAA region2.v2.argotunnel.com

; <<>> DiG 9.16.1-Ubuntu <<>> AAAA region2.v2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 350
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;region2.v2.argotunnel.com.   IN  AAAA

;; ANSWER SECTION:
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::9
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::10
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::7
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::8
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::2
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::1
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::3
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::6
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::5
region2.v2.argotunnel.com. 84908 IN   AAAA    2606:4700:a8::4

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jun 28 11:28:11 EDT 2023
;; MSG SIZE  rcvd: 334

**Environment and versions**
 - OS: Ubuntu
 - Architecture: Intel Arm docker run cloudflare/cloudflared --version
cloudflared version 2023.6.1 (built 2023-06-20-0923 UTC)
 - Version: [e.g. 2022.02.0]

Additional context I wonder if it's the docker container? Unfortunately, as this machine is managed, we can not run sudo for new code installs without a thorough and lengthy review.

same as you ,also run in container

raoinvoker commented 9 months ago

I'm having issues too. I get errors like this and then the hosted website just gives 502 errors perpetually until cloudflared is restarted. Very annoying. For now, I probably need a script to restart it every 3.5 hours or so.

me too, i run a nginx server exposed by cloudflare tunnel, and started a cron job to check if the nginx is accessable through cloudflare tunnel. if it fails, restart cloudflared.

Hey, have you tried running it on different network ? maybe try it with any other isp or maybe mobile hotspot ?

raoinvoker commented 9 months ago

I have the same issue with with cloudflared 2023.7.3-amd64, on a k3s cluster.

I can connect to the quic port using telnet or nc from inside the network just fine, but when running cloudflared, the container enters in CrashLoop because of this error.

RR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activit │
│ INF Retrying connection in up to 2s connIndex=0 event=0 ip=198.41.192.167                                            │
│ ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activit │
│ INF Retrying connection in up to 4s connIndex=0 event=0 ip=198.41.192.27                                             │
│ ERR Failed to create new quic connection error="failed to dial to edge with quic: timeout: no recent network activit │
│ INF Retrying connection in up to 8s connIndex=0 event=0 ip=198.41.192.77

hey, have you tried it with any other, network / isp, just try it with mobile network ?

codegax commented 4 months ago

Same issue with TrueCharts Clodflared container

PiotrCzapla commented 2 months ago

In my case the issue was caused by a stateless firewall that was not letting incoming UDP traffic to the cloudflared. Allowing all incoming traffic from cloudflare servers port 7488 to any port in ephemeral range (32768-65535) on my server fixed the issue.

If you want to test if this is the case on your server, try running an echo script on some server and then use nc to test your connection from docker host like this: nc <your-echo-server> 7844

Here is a sample echo server 

import socket
PORT=7844
server_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_socket.bind(('', PORT))  # Bind to port PORT on all interfaces

print(f"UDP server up and listening on port {PORT}")

while True:
    message, address = server_socket.recvfrom(1024)
    response = f"Message received from {address}: {repr(message)}\n"
    print(response)
    server_socket.sendto(response.encode(), address)

You should see something like this on your docker host if everything works okey. 

$ nc -u 65.108.76.43 7843
test
Message received from ('91.90.175.50', 59744): b'test\n'
test2
Message received from ('91.90.175.50', 59744): b'test2\n'

In my case until I've fixed firewall rules I wasn't getting any response back, even though my echo server was receiving messages.

Tanguille commented 2 weeks ago

I'm experiencing this aswell on ProxMox LXC built via Proxmox VE Helper-Scripts