search

cloudflare / cloudflared

Cloudflare Tunnel client (formerly Argo Tunnel)
https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-guide
Apache License 2.0
9.26k stars 820 forks source link

Tunnel stops working #593

Open JonasJasas opened 2 years ago

JonasJasas commented 2 years ago

Describe the bug The tunnel randomly stops working

Environment and versions Docker image cloudflare/cloudflared:2022.2.0

Logs and errors

2022-03-04T08:21:22Z INF Connection cfa57569-7451-44ef-aab4-5ee99096d6df registered connIndex=2 location=IAD
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=1
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=1
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=1
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=1
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=3
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=3
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=3
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=3                                                                                                                                                                                                                                                                                                                           2022-03-04T08:38:11Z INF Connection ac97da84-de5a-48b1-8adb-1aa61bac2386 registered connIndex=1 location=IAD
2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=2
2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=2
2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=2
2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=2
Recover from http2 response writer panic, error goroutine 943226123 [running]:
runtime/debug.Stack()
/usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write.func1()
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:236 +0x2f
panic({0xe8c700, 0x11bd190})
/usr/local/go/src/runtime/panic.go:1038 +0x215
golang.org/x/net/http2.(*responseWriter).write(0x11c9460, 0xc001493d10, {0xc000bc5000, 0x7ff67c061a98, 0x0}, {0x0, 0x7ff6a2eb4f18})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2708 +0x150
golang.org/x/net/http2.(*responseWriter).Write(0x0, {0xc000bc5000, 0xadb1e7, 0xc000707dc8})
/go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2697 +0x30
github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write(0xc000cf9380, {0xc000bc5000, 0x0, 0xdf})
/go/src/github.com/cloudflare/cloudflared/connection/http2.go:239 +0x64
zombiezen.com/go/capnproto2/rpc.(*streamTransport).SendMessage(0xc001493ce0, {0x11e4498, 0xc000cf93c0}, {{0xc000a1df50, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})
/go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:65 +0x149
github.com/cloudflare/cloudflared/tunnelrpc.(*transport).SendMessage(0xc000e8adc8, {0x11e4498, 0xc000cf93c0}, {{0xc000a1df50, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})                                                                                                                                                                                                                         /go/src/github.com/cloudflare/cloudflared/tunnelrpc/logtransport.go:28 +0x138                                                                                                                                                                                                                                                                                                             zombiezen.com/go/capnproto2/rpc.(*Conn).dispatchSend(0xc000cff600)                                                                                                                                                                                                                                                                                                                           
     /go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:104 +0x1b3                                                                                                                                                                                                                                                                                 
 created by zombiezen.com/go/capnproto2/rpc.NewConn                                                                                                                                                                                                                                                                                                                                           
     /go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/rpc.go:115 +0x29d                                                                                                                                                                                                                                                                                       
 Recover from http2 response writer panic, error goroutine 943226123 [running]:                                                                                                                                                                                                                                                                                                               
 runtime/debug.Stack()                                                                                                                                                                                                                                                                                                                                                                        
     /usr/local/go/src/runtime/debug/stack.go:24 +0x65                                                                                                                                                                                                                                                                                                                                        
 github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write.func1()                                                                                                                                                                                                                                                                                                                
     /go/src/github.com/cloudflare/cloudflared/connection/http2.go:236 +0x2f                                                                                                                                                                                                                                                                                                                  
 panic({0xe8c700, 0x11bd190})                                                                                                                                                                                                                                                                                                                                                                 
     /usr/local/go/src/runtime/panic.go:1038 +0x215                                                                                                                                                                                                                                                                                                                                           
 golang.org/x/net/http2.(*responseWriter).write(0x11c9460, 0xc001493d10, {0xc000bc5000, 0x7ff67aee25d0, 0x0}, {0x0, 0x7ff6a2eb4f18})                                                                                                                                                                                                                                                          
     /go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2708 +0x150                                                                                                                                                                                                                                                                                            
 golang.org/x/net/http2.(*responseWriter).Write(0x0, {0xc000bc5000, 0xadb1e7, 0xc000707dc8})                                                                                                                                                                                                                                                                                                  
     /go/src/github.com/cloudflare/cloudflared/vendor/golang.org/x/net/http2/server.go:2697 +0x30                                                                                                                                                                                                                                                                                             
 github.com/cloudflare/cloudflared/connection.(*http2RespWriter).Write(0xc000cf9380, {0xc000bc5000, 0x0, 0x35})                                                                                                                                                                                                                                                                               
     /go/src/github.com/cloudflare/cloudflared/connection/http2.go:239 +0x64                                                                                                                                                                                                                                                                                                                  
 zombiezen.com/go/capnproto2/rpc.(*streamTransport).SendMessage(0xc001493ce0, {0x11e4498, 0xc000cf93c0}, {{0xc00c794060, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})                                                                                                                                                                                                                          
     /go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:65 +0x149                                                                                                                                                                                                                                                                                  
 github.com/cloudflare/cloudflared/tunnelrpc.(*transport).SendMessage(0xc000e8adc8, {0x11e4498, 0xc000cf93c0}, {{0xc00c794060, 0x8, {0x8, 0x1}, 0xffffffffffffffff, 0x0}})                                                                                                                                                                                                                    
     /go/src/github.com/cloudflare/cloudflared/tunnelrpc/logtransport.go:28 +0x138                                                                                                                                                                                                                                                                                                            
 zombiezen.com/go/capnproto2/rpc.(*Conn).dispatchSend(0xc000cff600)                                                                                                                                                                                                                                                                                                                           
     /go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/transport.go:104 +0x1b3                                                                                                                                                                                                                                                                                 
 created by zombiezen.com/go/capnproto2/rpc.NewConn                                                                                                                                                                                                                                                                                                                                           
     /go/src/github.com/cloudflare/cloudflared/vendor/zombiezen.com/go/capnproto2/rpc/rpc.go:115 +0x29d                                                                                                                                                                                                                                                                                       
 2022-03-04T08:38:11Z INF Lost connection with the edge connIndex=0                                                                                                                                                                                                                                                                                                                           
 2022-03-04T08:38:11Z WRN Serve tunnel error error="connection with edge closed" connIndex=0                                                                                                                                                                                                                                                                                                  
 2022-03-04T08:38:11Z INF Retrying connection in up to 1s seconds connIndex=0                                                                                                                                                                                                                                                                                                                 
 2022-03-04T08:38:11Z INF Unregistered tunnel connection connIndex=0                                                                                                                                                                                                                                                                                                                          
 2022-03-04T08:38:12Z INF Connection 2a244dc7-b8e0-4342-a75f-9117c6130b23 registered connIndex=3 location=IAD                                                                                                                                                                                                                                                                                 
 2022-03-04T08:38:12Z INF Connection f616308f-6ca3-43e8-a599-c5ce3dea3f5a registered connIndex=2 location=IAD                                                                                                                                                                                                                                                                                 
 2022-03-04T08:38:12Z INF Connection 7c2787a1-f967-4d7e-875c-f27d39974d5f registered connIndex=0 location=IAD                                                                                                                                                                                                                                                                                 
 2022-03-04T08:38:31Z INF Lost connection with the edge connIndex=3                                                                                                                                                                                                                                                                                                                           
 2022-03-04T08:38:31Z WRN Serve tunnel error error="connection with edge closed" connIndex=3                                                                                                                                                                                                                                                                                                  
 2022-03-04T08:38:31Z INF Retrying connection in up to 1s seconds connIndex=3                                                                                                                                                                                                                                                                                                                 
 2022-03-04T08:38:31Z INF Unregistered tunnel connection connIndex=3                                                                                                                                                                                                                                                                                                                          
 2022-03-04T08:38:32Z INF Connection 686d1572-ddd6-4316-9b41-ee6ce6d54e6b registered connIndex=3 location=IAD                                                                                                                                                                                                                                                                                 
 2022-03-04T08:40:01Z INF Lost connection with the edge connIndex=3                                                                                                                                                                                                                                                                                                                           
 2022-03-04T08:40:01Z WRN Serve tunnel error error="connection with edge closed" connIndex=3                                                                                                                                                                                                                                                                                                  
 2022-03-04T08:40:01Z INF Retrying connection in up to 1s seconds connIndex=3

Additional context Running on Kubernetes 1.19.11 Docker version 20.10.7, build 20.10.7-0ubuntu5~18.04.3

savikko commented 2 years ago

I am not sure if this is related, but we are having problems with cloudflared and Kubernetes (DOKS) as well: Pod on kubernetes is creating tcp tunnel (running cloudflared access tcp..) to Windows host (running cloudflared tunnel ...).

It works fine about 30 seconds after start and then it starts getting errors: 2022-03-04T11:57:01Z ERR failed to connect to origin error="websocket: bad handshake" originURL=<REDACTED>

From client side this errors with ECONNRESET.

About 30-40% of connections succeeds.

Again, not sure if related at all.

At the same time another (windows) machine running cloudflared access tcp.. to same endpoint works 100%.

Running Kubernetes 1.20.15-do.0.