Too many open files

[mishamoix]

Hello 👋! Thanks for this library.

I'm building a vpn app for ios which based on xray. Well, I use your lib as a glue between raw packet data from the system and xray. But I faced with a problem that ios Network Extension (which dedicated for implementation core part of vpn) has memory constraint in 50mb. Had a problem that I exceed this limit on heavy websites like youtube.

I've tried setting a limit via limit-nofile: 500 and it works okay on basic websites but still has a problem on heavy ones.

In the logs, I frequently see session connect and client connect but in some moment I see an error failed to accepted raw connections > accept tcp [::1]:448: accept: too many open files, leading to websites not loading.

Could you provide any advice on improving memory consumption? Or is it possible for you to develop a patch that releases memory from unnecessary connections?

But thanks anyway for your titanic work!

P.S. With limit in 500 I still have a memory consumption around 40mb on iphone.

Example of log:

[2024-01-05 19:43:50] [E] 0x102728000 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x1040100d0 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x1040100d0 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x1040100d0 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x10400f050 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x10400f050 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x10400f050 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x10400f890 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x10400f890 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x10400f890 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x10400fe10 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x10400fe10 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x10400fe10 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x104010390 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x104010390 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x104010390 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x104010910 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x104010910 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x104010910 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x104010e90 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x104010e90 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x104010e90 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x104011150 socks5 client socket
[2024-01-05 19:43:50] [E] 0x104011150 socks5 client connect
[2024-01-05 19:43:50] [E] 0x104011150 socks5 session connect
2024/01/05 18:43:50 [Warning] transport/internet/tcp: failed to accepted raw connections > accept tcp [::1]:448: accept: too many open files
2024/01/05 18:43:50 [Warning] transport/internet/tcp: failed to accepted raw connections > accept tcp [::1]:448: accept: too many open files
2024/01/05 18:43:50 [Warning] transport/internet/tcp: failed to accepted raw connections > accept tcp [::1]:448: accept: too many open files
[2024-01-05 19:43:50] [E] 0x10271ee40 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x10271ee40 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x10271ee40 socks5 session handshake
[2024-01-05 19:43:50] [E] 0x10271f100 socks5 client udp socket
[2024-01-05 19:43:50] [E] 0x10271f100 socks5 client set upstream addr
[2024-01-05 19:43:50] [E] 0x10271f100 socks5 session handshake

2024/01/05 18:43:52 [Warning] transport/internet/tcp: failed to accepted raw connections > accept tcp [::1]:448: accept: too many open files```

[heiher]

Limiting only 500 open files is obviously not a reasonable solution. What I need to confirm is how much memory hev-socks5-tunnel will take up?

[mishamoix]

Not sure how you can test it locally, but I can see that memory consumption is growing without the 500 file limit.

[heiher]

No, I can't test it locally. Although I'm not an iOS developer, I know about the 50M memory limit (someone told me before). He also told me that hev-socks5-tunnel works effectively under that memory limit, which is why he switched from other projects. I think it is reasonable that memory usage increases as concurrent connections increase. The key point is whether the memory limit is exceeded in your user case?

[InstaRobot]

I have the same problem using this library. Is it possible to force the memory usage? Some sites open a huge number of connections. This is not very common, but it does happen.

[mishamoix]

It works effectively on 95% of websites, but some, who use a lot of ad trackers (my personal observation) the number of connections is growing rapidly (and it's okay and how it should work), but nice to have some settings to set max memory consumption or improve reusing of open connections. Don't know how I can help to you in this situation. If you have any idea, please tell me.

P.S. I've tried same websites on powerful iphones 14 and 15, it works very well, and clean the memory also good, so no exceed of 50mb. But still don't know why

[heiher]

Could you please paste the amount of memory actually used by hev-socks5-tunnel? I would like to know whether the 50M limit is only for the hev-socks5-tunnel process? Or does it also include xray.

The base library uses the slice memory allocator by default, which has a cache. Disabling it can reduce memory usage.

https://github.com/heiher/hev-task-system/blob/master/configs.mk#L6

ENABLE_MEMALLOC_SLICE := 0

[EbrahimTahernejad]

The 50MB memory limit is for Xray + Tun2Socks. And the moment you go above 50MB it'll crash so it's hard to assess how much it's using.

I suggest you use sing-box for now as it has the option to close connections when too much memory is being used.

[AndyLeeNH]

i meet the same problem. xcode shows too many udp sockets are opened. i'm using hevsocks+xray

xray can read the inbound and connect. but too many udp socket hanging there

[AndyLeeNH]

i meet the same problem. xcode shows too many udp sockets are opened. i'm using hevsocks+xray

xray can read the inbound and connect. but too many udp socket hanging there

if i change limit-nofile to 65535, the memory will soon be over 50MB

[heiher]

hev-socks5-tunnel only uses up to 14M of memory on the Android system, even though I have many web pages and applications open.

https://github.com/heiher/sockstun

For the constrained memory case, I think it would be possible to do LRU sortng of sessions, forcing eviction of the least active sessions when the session limit is exceeded.

Of course, the premise that this can solve the problem is that the memory is indeed used by hev-socks5-tunnel. Can anyone confirm this?

[AndyLeeNH]

hev-socks5-tunnel only uses up to 14M of memory on the Android system, even though I have many web pages and applications open.

https://github.com/heiher/sockstun

For the constrained memory case, I think it would be possible to do LRU sortng of sessions, forcing eviction of the least active sessions when the session limit is exceeded.

Of course, the premise that this can solve the problem is that the memory is indeed used by hev-socks5-tunnel. Can anyone confirm this?

大佬我可能冤枉你了, 看了下log, 也许是后端(同样在客户端的xray)给我的同一个udp请求返回了很多重复的包? 虽然, 这并不是确凿地说明了问题, 但也许是个提示

查看客户端xray的log, 也是收到很多回包, 有一些长度重复, 因此我怀疑是我的后端(服务器的xray)的问题