search

xjasonlyu / tun2socks

tun2socks - powered by gVisor TCP/IP stack
https://github.com/xjasonlyu/tun2socks/wiki
GNU General Public License v3.0
3.41k stars 464 forks source link

[Bug] tun2socks is getting killed with oom at the same time #415

Open fekazar opened 1 month ago

fekazar commented 1 month ago

Verify steps

Version

linux/amd64, go1.22.5, 488e5b2

What OS are you seeing the problem on?

Linux

Description

Tun2socks process is getting killed every day at about the same time with oom. I've tried scheduling the service restart every hour and decreasing buffers sizes, but the accident still occurs.

CLI or Config

My systemd unit file

[Unit]
Description=Tun2Socks
After=network.target

StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
MemoryMax=400M
MemoryAccounting=yes
Type=simple
User=root
Restart=on-failure
RestartSec=5s
EnvironmentFile=/etc/default/tun2socks
ExecStartPre=-ip tuntap add mode tun dev tun0
ExecStartPre=ip addr add 192.168.0.33/24 dev tun0
ExecStartPre=ip link set dev tun0 up
ExecStart=tun2socks -device tun://tun0 -proxy ss://chacha20-ietf-poly1305:${SSPASSWORD}@${SSIP}:${SSPORT} -loglevel silent -tcp-sndbuf 128k -tcp-rcvbuf 128k
ExecStartPost=bash -c 'MIP=$(ip r l |grep "default via" | cut -f3 -d" "); LIP=$(ip a l ens3 | awk \'/inet /{ print $2 }\' | cut -f1 -d"/"); ip r del default dev ens3; ip r add default via $MIP dev ens3 metric 200; ip rule add from $LIP table lip; ip r add default via $MIP dev ens3 table lip; ip r add ${SSIP}/32 via $MIP dev ens3'
ExecStartPost=ip r add default dev tun0 metric 50
ExecStopPost=-ip r flush table lip
ExecStopPost=-ip rule delete table lip
ExecStopPost=-ip link set dev tun0 down
ExecStopPost=-ip link del dev tun0
ExecStopPost=-ip r del ${SSIP}/32 dev ens3

Logs

Sep 18 08:21:44 weaselcloud-7511 kernel: Tasks state (memory values in pages):
Sep 18 08:21:44 weaselcloud-7511 kernel: memory: usage 409600kB, limit 409600kB, failcnt 1690
Sep 18 08:21:44 weaselcloud-7511 kernel:  mem_cgroup_out_of_memory+0x13f/0x160
Sep 18 08:21:44 weaselcloud-7511 kernel:  out_of_memory+0x106/0x2e0
Sep 17 08:22:18 weaselcloud-7511 kernel: Memory cgroup out of memory: Killed process 342254 (tun2socks) total-vm:1634192kB, anon-rss:404808kB, file-rss:4800kB, shmem-rss:0kB, UID:0 pgtables:960kB oom_score_adj:0
Sep 17 08:22:17 weaselcloud-7511 kernel: Tasks state (memory values in pages):
Sep 17 08:22:17 weaselcloud-7511 kernel: memory: usage 409600kB, limit 409600kB, failcnt 290
Sep 17 08:22:17 weaselcloud-7511 kernel:  mem_cgroup_out_of_memory+0x13f/0x160
Sep 17 08:22:17 weaselcloud-7511 kernel:  out_of_memory+0x106/0x2e0
Sep 16 08:27:56 weaselcloud-7511 kernel: Memory cgroup out of memory: Killed process 332131 (tun2socks) total-vm:1697244kB, anon-rss:400412kB, file-rss:6524kB, shmem-rss:0kB, UID:0 pgtables:1036kB oom_score_adj:0
Sep 16 08:27:56 weaselcloud-7511 kernel: Tasks state (memory values in pages):
Sep 16 08:27:55 weaselcloud-7511 kernel: memory: usage 409600kB, limit 409600kB, failcnt 1374
Sep 16 08:27:54 weaselcloud-7511 kernel:  mem_cgroup_out_of_memory+0x13f/0x160
Sep 16 08:27:54 weaselcloud-7511 kernel:  out_of_memory+0x106/0x2e0
Sep 15 08:22:22 weaselcloud-7511 kernel: Memory cgroup out of memory: Killed process 324855 (tun2socks) total-vm:1831004kB, anon-rss:398160kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1292kB oom_score_adj:0
Sep 15 08:22:22 weaselcloud-7511 kernel: Tasks state (memory values in pages):
Sep 15 08:22:22 weaselcloud-7511 kernel: memory: usage 409600kB, limit 409600kB, failcnt 193075
Sep 15 08:22:21 weaselcloud-7511 kernel:  mem_cgroup_out_of_memory+0x13f/0x160
Sep 15 08:22:21 weaselcloud-7511 kernel:  out_of_memory+0x106/0x2e0
Sep 14 08:21:23 weaselcloud-7511 kernel: Memory cgroup out of memory: Killed process 312411 (tun2socks) total-vm:1896032kB, anon-rss:398308kB, file-rss:564kB, shmem-rss:0kB, UID:0 pgtables:1400kB oom_score_adj:0
Sep 14 08:21:23 weaselcloud-7511 kernel: Tasks state (memory values in pages):
Sep 14 08:21:23 weaselcloud-7511 kernel: memory: usage 409600kB, limit 409600kB, failcnt 25235
Sep 14 08:21:23 weaselcloud-7511 kernel:  mem_cgroup_out_of_memory+0x13f/0x160
Sep 14 08:21:23 weaselcloud-7511 kernel:  out_of_memory+0x106/0x2e0
Sep 13 08:21:32 weaselcloud-7511 kernel: Memory cgroup out of memory: Killed process 302766 (tun2socks) total-vm:1698124kB, anon-rss:403248kB, file-rss:724kB, shmem-rss:0kB, UID:0 pgtables:1020kB oom_score_adj:0
Sep 13 08:21:32 weaselcloud-7511 kernel: Tasks state (memory values in pages):
Sep 13 08:21:32 weaselcloud-7511 kernel: memory: usage 409600kB, limit 409600kB, failcnt 10715
Sep 13 08:21:31 weaselcloud-7511 kernel:  mem_cgroup_out_of_memory+0x13f/0x160
Sep 13 08:21:31 weaselcloud-7511 kernel:  out_of_memory+0x106/0x2e0
Sep 12 08:21:38 weaselcloud-7511 kernel: Memory cgroup out of memory: Killed process 285292 (tun2socks) total-vm:1764888kB, anon-rss:401016kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1240kB oom_score_adj:0

How to Reproduce

I suppose launching the attached config is enough.

xjasonlyu commented 1 month ago

Have you tried increasing the memory limit to see if it gets better? I'm not sure if it's due to usage or a real memory leak. 🤔

fekazar commented 1 month ago

When there was no memory limit for the service the whole system could freeze and I had to reboot the server. With this limit it can recover by itself. I grep the memory usage from systemctl every 5 minutes, here is that simple log, it doesn't look like there is a leak.

I also have a service restart at every 33-rd minute in crontab. 

     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 1min 57s ago
     Memory: 1.5M (max: 400.0M available: 398.4M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 6min ago
     Memory: 1.5M (max: 400.0M available: 398.4M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 11min ago
     Memory: 10.9M (max: 400.0M available: 389.0M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 16min ago
     Memory: 8.3M (max: 400.0M available: 391.6M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 21min ago
     Memory: 8.0M (max: 400.0M available: 391.9M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 26min ago
     Memory: 7.3M (max: 400.0M available: 392.6M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 31min ago
     Memory: 9.7M (max: 400.0M available: 390.2M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 36min ago
     Memory: 29.1M (max: 400.0M available: 370.8M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 41min ago
     Memory: 16.6M (max: 400.0M available: 383.3M)
     Active: active (running) since Wed 2024-09-18 07:33:04 UTC; 46min ago
     Memory: 20.5M (max: 400.0M available: 379.4M)
     Active: active (running) since Wed 2024-09-18 08:21:57 UTC; 3min 4s ago
     Memory: 25.3M (max: 400.0M available: 374.6M)
     Active: active (running) since Wed 2024-09-18 08:21:57 UTC; 8min ago
     Memory: 25.9M (max: 400.0M available: 374.0M)
     Active: active (running) since Wed 2024-09-18 08:33:05 UTC; 1min 56s ago
     Memory: 18.2M (max: 400.0M available: 381.7M)
     Active: active (running) since Wed 2024-09-18 08:33:05 UTC; 6min ago
     Memory: 10.4M (max: 400.0M available: 389.5M)
     Active: active (running) since Wed 2024-09-18 08:33:05 UTC; 11min ago
     Memory: 19.4M (max: 400.0M available: 380.5M)