rapiz1 / rathole

A lightweight and high-performance reverse proxy for NAT traversal, written in Rust. An alternative to frp and ngrok.
Apache License 2.0
9.8k stars 488 forks source link

Failed to write control cmds: Broken pipe (os error 32) #148

Closed flo82 closed 2 years ago

flo82 commented 2 years ago

Describe the bug

After a couple of time this error occurs suddenly

To Reproduce wait.

A Client restart will fix this issue.

Configuration Configuration used to reproduce the behavior:

Server

# server.toml
[server]
bind_addr = "0.0.0.0:2333" # `2333` specifies the port that rathole listens for clients
default_token = "xxx" # Must be the same with the server to pass the validation

[server.services.xxx_udp]
bind_addr = "0.0.0.0:28914" # The address of the service that needs to be forwarded
type = "udp"

[server.services.xxx_tcp]
bind_addr = "0.0.0.0:28914" # The address of the service that needs to be forwarded
type = "tcp"

Client

# client.toml
[client]
remote_addr = "xxx:2333" # The address of the server. The port must be the same with the port in `server.bind_addr`
default_token = "xxx" # Must be the same with the server to pass the validation

[client.services.xxx_udp]
local_addr = "xxx:28967" # The address of the service that needs to be forwarded
type = "udp"

[client.services.xxx_tcp]
local_addr = "xxx:28967" # The address of the service that needs to be forwarded
type = "tcp"

Logs server:

Mar 28 05:09:47 xxx.host rathole[2802300]: Mar 28 05:09:47.076 ERROR connection{addr=x.x.x.x:47474}:handle{service=xxx_tcp}:run: rathole::server: Failed to write control cmds: Broken pipe (os error 32)
Mar 28 05:09:47 xxx.host rathole[2802300]: Mar 28 05:09:47.076  INFO connection{addr=x.x.x.x:47474}:handle{service=xxx_tcp}:run: rathole::server: Control channel shutdown
Mar 28 05:09:47 xxx.host rathole[2802300]: Mar 28 05:09:47.731  INFO connection{addr=x.x.x.x:47474}:handle{service=xxx_tcp}:run_tcp_connection_pool: rathole::server: TCPListener shutdown
Mar 28 05:09:49 xxx.host rathole[2802300]: Mar 28 05:09:49.103 ERROR connection{addr=x.x.x.x:47472}:handle{service=xxx_udp}:run: rathole::server: Failed to write control cmds: Broken pipe (os error 32)
Mar 28 05:09:49 xxx.host rathole[2802300]: Mar 28 05:09:49.103  INFO connection{addr=x.x.x.x:47472}:handle{service=xxx_udp}:run: rathole::server: Control channel shutdown

client

Mar 28 09:08:49.171 DEBUG handle{service=xxx_udp}:run: rathole::client: Received HeartBeat
Mar 28 09:09:09.634  WARN handle{service=xxx_tcp}:run: rathole::client: Heartbeat timed out
Mar 28 09:09:09.634  INFO handle{service=xxx_tcp}:run: rathole::client: Control channel shutdown
Mar 28 09:09:18.091 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:23.331 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:25.497 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:25.559 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:25.619 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:26.164 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:28.799 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:29.172  WARN handle{service=xxx_udp}:run: rathole::client: Heartbeat timed out
Mar 28 09:09:29.172  INFO handle{service=xxx_udp}:run: rathole::client: Control channel shutdown
Mar 28 09:09:34.394 DEBUG handle{service=xxx_tcp}:run:run_data_channel_for_tcp{local_addr="xxx:28967"}: rathole::client: New data channel starts forwarding
Mar 28 09:09:50.638 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:10:51.233 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:11:05.479 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:11:23.472 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:11:59.590 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:12:06.468 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:12:24.024 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:12:52.334 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:13:00.596 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:14:03.845 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:15:25.524 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:16:22.362 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:16:35.544 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:16:47.899 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:17:24.186 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped
Mar 28 09:17:39.684 DEBUG run_udp_forwarder: rathole::client: Forwarder created
Mar 28 09:17:49.893 DEBUG run_udp_forwarder: rathole::client: Forwarder dropped

ignore time differecnes - different timezones

Environment:

rathole
Build Timestamp:     2022-03-09T04:41:52.409645741+00:00
Build Version:       0.4.0
Commit SHA:          Some("952c4134c3e3e64fd0e479fa7327abb3e9bea0f3")
Commit Date:         Some("2022-03-09T04:34:45+00:00")
Commit Branch:       Some("detached HEAD")
cargo Target Triple: x86_64-unknown-linux-gnu
cargo Profile:       release
cargo Features:      base64,client,default,hot_reload,noise,notify,server,snowstorm,tls,tokio_native_tls
rapiz1 commented 2 years ago

Network loss can cause these errors and it's expected to recover by the client reconnecting. Possibly dup of #145 and #143 ( issues in Chinese)

A fix has already been made. Can you try https://github.com/rapiz1/rathole/actions/runs/2038127693 to see if it fix the issue?

flo82 commented 2 years ago

are there any docker builds because my client is using docker....

Otherwise I'm waiting for the next official release.

Thanks for the quick response.

rapiz1 commented 2 years ago

You can try docker now

flo82 commented 2 years ago

Lgtm. Thank you