search

cloudflare / cloudflared

Cloudflare Tunnel client (formerly Argo Tunnel)
https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide
Apache License 2.0
9.19k stars 809 forks source link

🐛 there are no free edge addresses left to resolve to #1060

Open oraant opened 1 year ago

oraant commented 1 year ago

Describe the bug

Tunnel connector start failed with error: "there are no free edge addresses left to resolve to"

To Reproduce Steps to reproduce the behavior:

  1. Configure 'Follow the official website guide to create tunnel'
  2. Run 'docker run cloudflare/cloudflared:latest tunnel --no-autoupdate run --token xxx'
  3. See error: there are no free edge addresses left to resolve to

If it's an issue with Cloudflare Tunnel:

  1. Tunnel ID : f045ed83-0fcb-4c7d-8c75-9fc74d6bd0d8
  2. cloudflared config:

Expected behavior

I tried install the cloudflared with dpkg / docker / synology package. I tried create tunnel with dashboard / cli command I tried reboot the synology / router I tried different version of cloudflared package.

But, I get this error every time.

The firewall on the synology is closed. I don't know what else I can do now.

I just hope it can normally run.

Environment and versions

Logs and errors

root@OraAnt-Ds920p:~# docker run cloudflare/cloudflared:latest tunnel  --loglevel debug --no-autoupdate run --token xxxxxxxx
2023-08-31T15:32:03Z INF Starting tunnel tunnelID=f045ed83-0fcb-4c7d-8c75-9fc74d6bd0d8
2023-08-31T15:32:03Z INF Version 2023.8.1
2023-08-31T15:32:03Z INF GOOS: linux, GOVersion: go1.20.6, GoArch: amd64
2023-08-31T15:32:03Z INF Settings: map[loglevel:debug no-autoupdate:true token:*****]
2023-08-31T15:32:03Z INF Generated Connector ID: 079a44d4-092a-488c-9235-cf7165f9b9b1
2023-08-31T15:32:03Z DBG Fetched protocol: quic
2023-08-31T15:32:03Z INF Initial protocol quic
2023-08-31T15:32:03Z INF ICMP proxy will use 172.17.0.4 as source for IPv4
2023-08-31T15:32:03Z DBG Failed to determine the IPv6 for this machine. It will use :: to send/listen for ICMPv6 echo
2023-08-31T15:32:03Z INF ICMP proxy will use :: as source for IPv6
2023-08-31T15:32:03Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 65532 is not between ping group 1 to 0"
2023-08-31T15:32:03Z DBG ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2023-08-31T15:32:03Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 65532 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2023-08-31T15:32:03Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
2023-08-31T15:32:03Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a0::10","2606:4700:a0::9","2606:4700:a0::8","2606:4700:a0::3","2606:4700:a0::2","2606:4700:a0::5","2606:4700:a0::1","2606:4700:a0::7","2606:4700:a0::4","2606:4700:a0::6"] event=0
2023-08-31T15:32:03Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a8::6","2606:4700:a8::9","2606:4700:a8::10","2606:4700:a8::2","2606:4700:a8::1","2606:4700:a8::8","2606:4700:a8::4","2606:4700:a8::5","2606:4700:a8::3","2606:4700:a8::7"] event=0
2023-08-31T15:32:03Z INF Starting metrics server on 127.0.0.1:35160/metrics
2023-08-31T15:32:03Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
2023-08-31T15:32:03Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a0::5","2606:4700:a0::1","2606:4700:a0::7","2606:4700:a0::4","2606:4700:a0::6","2606:4700:a0::10","2606:4700:a0::9","2606:4700:a0::8","2606:4700:a0::3","2606:4700:a0::2"] event=0
2023-08-31T15:32:03Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a8::5","2606:4700:a8::3","2606:4700:a8::7","2606:4700:a8::6","2606:4700:a8::9","2606:4700:a8::10","2606:4700:a8::2","2606:4700:a8::1","2606:4700:a8::8","2606:4700:a8::4"] event=0
2023-08-31T15:32:03Z INF You requested 4 HA connections but I can give you at most 0.
2023-08-31T15:32:03Z DBG edge discovery: no addresses left in pool to give proxy connection connIndex=0 event=0
2023-08-31T15:32:03Z INF Tunnel server stopped
2023-08-31T15:32:03Z ERR Initiating shutdown error="there are no free edge addresses left to resolve to"
2023-08-31T15:32:04Z WRN Your version 2023.8.1 is outdated. We recommend upgrading it to 2023.8.2
2023-08-31T15:32:04Z INF Metrics server stopped
there are no free edge addresses left to resolve to
root@OraAnt-Ds920p:~#
nikotung commented 1 year ago

I also encounter this issue.

Environment and versions

2023-09-12T13:35:02Z INF Starting tunnel tunnelID=f6a0bf44-b65e-4dd2-bd0e-666ed08e9c6a
2023-09-12T13:35:02Z INF Version 2023.8.2
2023-09-12T13:35:02Z INF GOOS: linux, GOVersion: go1.20.6, GoArch: amd64
2023-09-12T13:35:02Z INF Settings: map[cred-file:/home/niko/.cloudflared/f6a0bf44-b65e-4dd2-bd0e-666ed08e9c6a.json credentials-file:/home/niko/.cloudflared/f6a0bf44-b65e-4dd2-bd0e-666ed08e9c6a.json]
2023-09-12T13:35:02Z INF cloudflared will not automatically update if installed by a package manager.
2023-09-12T13:35:02Z INF Generated Connector ID: 15ccf66c-d50a-4f60-9078-5d07c001ecb5
2023-09-12T13:35:02Z INF Initial protocol quic
2023-09-12T13:35:02Z INF ICMP proxy will use 192.168.8.169 as source for IPv4
2023-09-12T13:35:02Z INF ICMP proxy will use fd88:3d88:8354::b20 in zone enp2s0 as source for IPv6
2023-09-12T13:35:02Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 1000 is not between ping group 1 to 0"
2023-09-12T13:35:02Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2023-09-12T13:35:02Z INF Starting metrics server on 127.0.0.1:42409/metrics
2023-09-12T13:35:02Z INF You requested 4 HA connections but I can give you at most 0.
2023-09-12T13:35:02Z INF Tunnel server stopped
2023-09-12T13:35:02Z ERR Initiating shutdown error="there are no free edge addresses left to resolve to"
2023-09-12T13:35:02Z INF Metrics server stopped
there are no free edge addresses left to resolve to
nikotung commented 1 year ago

It seems this issue can be solved by parsing the argotunnel.com domain to A type not AAAA. After I forced the argotunnel.com with A type during DNS resolve, the tunnel can run up.

not working with AAAA type domain argotunnel.com .

2023-09-13T13:21:48Z INF Settings: map[cred-file:/home/niko/.cloudflared/f6a0bf44-b65e-4dd2-bd0e-666ed08e9c6a.json credentials-file:/home/niko/.cloudflared/f6a0bf44-b65e-4dd2-bd0e-666ed08e9c6a.json loglevel:debug]                 [137/1970]
2023-09-13T13:21:48Z INF cloudflared will not automatically update if installed by a package manager.                                                                                                                                           
2023-09-13T13:21:48Z INF Generated Connector ID: 0b4a179a-435d-49cd-a6c8-a6a6b6f7bfba                                                                                                                                                           
2023-09-13T13:21:48Z DBG Refreshed feature account_hash=62 pq_enabled=false pq_perct=5                                                                                                                                                          
2023-09-13T13:21:48Z DBG Fetched protocol: quic                                                                                                                                                                                                 
2023-09-13T13:21:48Z INF Initial protocol quic                                                                                                                                                                                                  
2023-09-13T13:21:48Z INF ICMP proxy will use 192.168.8.169 as source for IPv4                                                                                                                                                                   
2023-09-13T13:21:48Z INF ICMP proxy will use fd88:3d88:8354:0:6b5e:35f6:9b4c:9d86 in zone enp2s0 as source for IPv6                                                                                                                             
2023-09-13T13:21:48Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user 
is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 1000 is not between ping group 1 to 0"                                                              
2023-09-13T13:21:48Z DBG ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"                                                          
2023-09-13T13:21:48Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"                                                          
2023-09-13T13:21:48Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0                                                                                                                        
2023-09-13T13:21:48Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a0::10","2606:4700:a0::7","2606:4700:a0::5","2606:4700:a0::8","2606:4700:a0::3","2606:4700:a0::6","2606:4700:a0::4","2606:4700:a0::9","2606:4700:a0::1","
2606:4700:a0::2"] event=0                                                                                                                                                                                                                       
2023-09-13T13:21:48Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a8::9","2606:4700:a8::7","2606:4700:a8::5","2606:4700:a8::10","2606:4700:a8::1","2606:4700:a8::8","2606:4700:a8::2","2606:4700:a8::3","2606:4700:a8::4","
2606:4700:a8::6"] event=0                                                                                                                                                                                                                       
2023-09-13T13:21:48Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0                                                                                                                        
2023-09-13T13:21:48Z INF Starting metrics server on 127.0.0.1:38209/metrics                                                                                                                                                                     
2023-09-13T13:21:48Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a0::10","2606:4700:a0::7","2606:4700:a0::5","2606:4700:a0::8","2606:4700:a0::3","2606:4700:a0::6","2606:4700:a0::4","2606:4700:a0::9","2606:4700:a0::1","
2606:4700:a0::2"] event=0                                                                                                                                                                                                                       
2023-09-13T13:21:48Z DBG edge discovery: resolved edge addresses addresses=["2606:4700:a8::9","2606:4700:a8::7","2606:4700:a8::5","2606:4700:a8::10","2606:4700:a8::1","2606:4700:a8::8","2606:4700:a8::2","2606:4700:a8::3","2606:4700:a8::4","
2606:4700:a8::6"] event=0                                                                                                                                                                                                                       
2023-09-13T13:21:48Z INF You requested 4 HA connections but I can give you at most 0.                                                                                                                                                           
2023-09-13T13:21:48Z DBG edge discovery: no addresses left in pool to give proxy connection connIndex=0 event=0                                                                                                                                 
2023-09-13T13:21:48Z INF Tunnel server stopped                                                                                                                                                                                                  
2023-09-13T13:21:48Z ERR Initiating shutdown error="there are no free edge addresses left to resolve to"                                                                                                                                        
2023-09-13T13:21:49Z INF Metrics server stopped                                                                                                                                                                                                 
there are no free edge addresses left to resolve to

working with A type domain argotunnel.com

2023-09-13T13:46:20Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user 
is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 1000 is not between ping group 1 to 0"
2023-09-13T13:46:20Z DBG ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2023-09-13T13:46:20Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2023-09-13T13:46:20Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
2023-09-13T13:46:20Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.57","198.41.192.67","198.41.192.37","198.41.192.107","198.41.192.47","198.41.192.227","198.41.192.167","198.41.192.7","198.41.192.77","198.41.192.27"] e
vent=0
2023-09-13T13:46:20Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.53","198.41.200.13","198.41.200.113","198.41.200.63","198.41.200.73","198.41.200.43","198.41.200.33","198.41.200.193","198.41.200.233","198.41.200.23"] 
event=0
2023-09-13T13:46:20Z INF Starting metrics server on 127.0.0.1:41197/metrics
2023-09-13T13:46:20Z DBG edge discovery: looking up edge SRV record domain=_v2-origintunneld._tcp.argotunnel.com event=0
2023-09-13T13:46:21Z DBG edge discovery: resolved edge addresses addresses=["198.41.192.67","198.41.192.37","198.41.192.107","198.41.192.47","198.41.192.227","198.41.192.167","198.41.192.7","198.41.192.77","198.41.192.27","198.41.192.57"] e
vent=0
2023-09-13T13:46:21Z DBG edge discovery: resolved edge addresses addresses=["198.41.200.13","198.41.200.113","198.41.200.63","198.41.200.73","198.41.200.43","198.41.200.33","198.41.200.193","198.41.200.233","198.41.200.23","198.41.200.53"] 
event=0
2023-09-13T13:46:21Z DBG edge discovery: giving new address to connection connIndex=0 event=0 ip=198.41.192.77
2023/09/13 21:46:21 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2023-09-13T13:46:21Z DBG QUIC TLS event curve=p256 handshake=true handshake_duration=6.539419ms
2023-09-13T13:46:22Z INF Registered tunnel connection connIndex=0 connection=18f44734-d5cd-4962-8e3f-dcce20f77ff8 event=0 ip=198.41.192.77 location=hkg09 protocol=quic
2023-09-13T13:46:22Z DBG edge discovery: giving new address to connection connIndex=1 event=0 ip=198.41.200.33
2023-09-13T13:46:22Z DBG QUIC TLS event curve=p256 handshake=true handshake_duration=4.958137ms
2023-09-13T13:46:23Z DBG edge discovery: giving new address to connection connIndex=2 event=0 ip=198.41.192.57
2023-09-13T13:46:23Z INF Registered tunnel connection connIndex=1 connection=9997ae1d-f1e1-4cd6-917b-324a0510647a event=0 ip=198.41.200.33 location=sin08 protocol=quic
2023-09-13T13:46:23Z DBG QUIC TLS event curve=p256 handshake=true handshake_duration=4.898445ms
2023-09-13T13:46:24Z INF Registered tunnel connection connIndex=2 connection=361624a7-14c3-4a26-a2ca-30636edc6e99 event=0 ip=198.41.192.57 location=hkg08 protocol=quic
2023-09-13T13:46:24Z DBG edge discovery: giving new address to connection connIndex=3 event=0 ip=198.41.200.113
2023-09-13T13:46:25Z DBG QUIC TLS event curve=p256 handshake=true handshake_duration=5.316791ms
2023-09-13T13:46:26Z INF Registered tunnel connection connIndex=3 connection=7a811070-1481-472b-a8d2-7b2565630071 event=0 ip=198.41.200.113 location=sin07 protocol=quic
microzoa commented 1 year ago

I'm receiving the same issue as OP, @NikoTung can you explain how you resolved this with "After I forced the argotunnel.com with A type during DNS resolve" - thanks

nikotung commented 1 year ago

I'm receiving the same issue as OP, @NikoTung can you explain how you resolved this with "After I forced the argotunnel.com with A type during DNS resolve" - thanks

Because my network support ipv6, at first it resolved the argotunnel.com domain to a ipv6 IP which causes the problem there is no free edge address left to.... I just try to force this domain with a ipv4 IP, and it works.

KleinPan commented 4 months ago

when i delete the AAAA record，it still appear Initiating shutdown error="there are no free edge addresses left to resolve to"，someone can help me? i run it in a NAS's docker.

harveyjing commented 1 month ago

I also have this problem