Cannot set UDP server recv buffer: No buffer space available in MacBook M1

[LesenHuang]

./wstunnel client -L 'udp://51280:localhost:51280?timeout_sec=0' wss://server.com:sport 2024-01-19T07:47:40.874012Z INFO wstunnel::udp: Starting UDP server listening cnx on 127.0.0.1:51280 with cnx timeout of 0s 2024-01-19T07:47:40.874215Z WARN wstunnel::udp: Cannot set UDP server recv buffer: No buffer space available (os error 55) 2024-01-19T07:47:40.874221Z WARN wstunnel::udp: Cannot set UDP server recv buffer: No buffer space available (os error 55)

[erebe]

Hello,

I will make a fix next week regarding that. I think if you start wstunnel with root/sudo, it should be good.

[erebe]

You may want to try this new release, I improved the buffer allocation strategy https://github.com/erebe/wstunnel/releases/tag/v9.2.0

[erebe]

Feel to re-open if needed :)

[LesenHuang]

sudo ./wstunnel client -L 'udp://51280:localhost:51280?timeout_sec=0' wss://serverip:sport 2024-01-22T09:37:40.950567Z INFO wstunnel::udp: Starting UDP server listening cnx on 127.0.0.1:51280 with cnx timeout of 0s 2024-01-22T09:37:40.950651Z WARN wstunnel::udp: Cannot increase UDP server recv buffer to 64 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950657Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950659Z WARN wstunnel::udp: Cannot increase UDP server recv buffer to 32 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950661Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950663Z WARN wstunnel::udp: Cannot increase UDP server recv buffer to 16 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950664Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950666Z WARN wstunnel::udp: Cannot increase UDP server recv buffer to 8 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950668Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950670Z INFO wstunnel::udp: Increased UDP server recv buffer to 4 Mib 2024-01-22T09:37:40.950672Z WARN wstunnel::udp: Cannot increase UDP server send buffer to 64 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950801Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950807Z WARN wstunnel::udp: Cannot increase UDP server send buffer to 32 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950810Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950812Z WARN wstunnel::udp: Cannot increase UDP server send buffer to 16 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950814Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950889Z WARN wstunnel::udp: Cannot increase UDP server send buffer to 8 Mib: No buffer space available (os error 55) 2024-01-22T09:37:40.950892Z WARN wstunnel::udp: This is not fatal, but can lead to packet loss if you have too much throughput. You must monitor packet loss in this case 2024-01-22T09:37:40.950894Z INFO wstunnel::udp: Increased UDP server send buffer to 4 Mib ^C

[LesenHuang]

The problem doesn't seem to be solved, there are a few more warning logs.

[erebe]

That's fine :) That's the expected behavior.

Before it would just fail to increase the buffer size, leaving it tiny. Now it tries various size in decrement, until one is accepted by your OS.

You are good to go. If you need to increase it more (should be good) you must change the kernel param https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

[LesenHuang]

The server uses centos7 + wstunnel_9.2.0_linux_amd64.tar.gz without warning. There is a warning when the client uses macos M1 + wstunnel_9.2.0_darwin_arm64.tar.gz

[erebe]

Yes, this is because Linux and Darwin/MacOs does not have the same behavior regarding buffers.

Linux say ok, but allocate memory on the fly/lazilly when the demands increase. MacOs allocate the memory upfront, and refuse to allocate more than the max.

Again you should be fine with a 4Mb buffer, if this is not the case, you must increase what you kernel allows for Mac https://medium.com/@CameronSparr/increase-os-udp-buffers-to-improve-performance-51d167bb1360

[LesenHuang]

Open port 51820 on the server side and cannot check the data from the wstunnel port. Operation case centos7 server: wstunnel server --restrict-to localhost:51820 wss://[::]:443 ncat -ln 51820 No data here

mac client: wstunnel client -L 'udp://51280:localhost:51280?timeout_sec=0' wss://my.server.com:443 curl my.server.com:51820 -d "hello, wstunnel"

[LesenHuang]

Is this buffer warning the reason why data cannot be received?

[erebe]

Your setup is not correct.

Your netcat and curl are using TCP, but your tunnel use udp://, so nothing is going to happen.

Tries like that.

wstunnel server --restrict-to localhost:51820 wss://[::]:443
ncat -ln 51820

client

wstunnel client -L 'tcp://51280:localhost:51280' wss://my.server.com:443
curl http://localhost:51820 -d "hello, wstunnel"

[LesenHuang]

I've corrected it and it's ready to go. But I want to test udp, how to test

[LesenHuang]

Because wireguard is used, it uses udp protocol. The following is an error that occurs in wstunnel after connecting with wireguard.

[LesenHuang]

2024-01-22T10:21:04.161669Z  INFO wstunnel::udp: New UDP connection from 127.0.0.1:49511
2024-01-22T10:21:04.161732Z  INFO wstunnel::tcp: Opening TCP connection to my.server.com:1002
2024-01-22T10:21:04.200044Z  INFO wstunnel::tls: Doing TLS handshake using sni IpAddress(my.server.com) with the server my.server.com:1002
2024-01-22T10:21:34.164333Z ERROR tunnel{id="018d30b0-1b21-7d1f-b6b0-dfc66c479803" remote="localhost:51280"}: wstunnel::tunnel::client: failed to get a connection to the server from the pool: TimedOut
2024-01-22T10:21:34.164413Z  INFO wstunnel::udp: New UDP connection from 127.0.0.1:49511
2024-01-22T10:21:34.164493Z  INFO wstunnel::tcp: Opening TCP connection to my.server.com:1002
2024-01-22T10:21:44.166191Z  WARN wstunnel::tcp: Cannot connect to tcp endpoint my.server.com:1002 due to timeout of 10s elapsed
2024-01-22T10:21:44.568210Z  INFO wstunnel::tcp: Opening TCP connection to my.server.com:1002
2024-01-22T10:21:54.570781Z  WARN wstunnel::tcp: Cannot connect to tcp endpoint my.server.com:1002 due to timeout of 10s elapsed
2024-01-22T10:21:55.372972Z  INFO wstunnel::tcp: Opening TCP connection to my.server.com:1002
2024-01-22T10:21:58.524764Z ERROR tunnel{id="018d30b0-9054-7552-ab8e-e4eb144acaab" remote="localhost:51280"}: wstunnel::tunnel::client: failed to get a connection to the server from the pool: User(failed to do TLS handshake with the server my.server.com:1002