search

SagerNet / sing-box

The universal proxy platform
https://sing-box.sagernet.org/
Other
16.6k stars 2.02k forks source link

Cannot close connections properly #1245

Closed firedes closed 7 months ago

firedes commented 7 months ago

Operating system

Linux

System version

Openwrt 18.06

Installation type

Original sing-box Command Line

If you are using a graphical client, please provide the version of the client.

No response

Version

1.7.5

Description

Seems sing-box client does not close stale connections properly, would potentially exhaust memory. (In my case, memory consumption increases over time) image image

Reproduction

Just run sing-box normally

Logs

No response

aur3l14no commented 7 months ago

Care to post config? I don't think it's reproducible for most people.

firedes commented 7 months ago

@aur3l14no Of course. 192.168.31.1 is my router's ip, 192.168.31.108 is my NAS ip, on which a mosdns is running.

{
  "log": {
    "disabled": false,
    "level": "info",
    "output": "box.log",
    "timestamp": true
  },
  "experimental": {
    "clash_api": {
      "external_controller": "192.168.31.1:9999",
      "external_ui": "yacd",
      "external_ui_download_url": "http://192.168.31.108:8080/download/sbox/gh-pages.zip",
      "external_ui_download_detour": "lan-out",
      "default_mode": "Rule"
    }
  },
  "dns": {
    "servers": [
      {
        "tag": "lan-dns",
        "address": "192.168.31.108",
        "detour": "lan-out"
      },
      {
        "tag": "refuse",
        "address": "rcode://name_error"
      }
    ],
    "final": "lan-dns",
    "strategy": "ipv4_only"
  },
  "inbounds": [
    {
      "type": "redirect",
      "tag": "redirect-in",
      "listen": "::",
      "listen_port": 7891,
      "sniff": true,
      "sniff_override_destination": false
    },
    {
      "type": "direct",
      "tag": "dns-in",
      "listen": "::",
      "listen_port": 7890,
      "network": "udp"
    },
    {
      "type": "tun",
      "tag": "tun-in",
      "interface_name": "tun0",
      "inet4_address": "172.19.0.1/30",
      "mtu": 9000,
      "auto_route": false,
      "strict_route": false,
      "stack": "system",
      "sniff": true,
      "sniff_override_destination": false
    }
  ],
  "outbounds": [
    {
      "type": "vless",
      "tag": "proxy-out",
      "server": "*",
      "server_port": 443,
      "uuid": "uuid",
      "flow": "xtls-rprx-vision",
      "tls": {
        "enabled": true,
        "disable_sni": false,
        "server_name": "server_name",
        "insecure": false,
        "utls": {
          "enabled": true,
          "fingerprint": "chrome"
        },
        "reality": {
          "enabled": true,
          "public_key": "public_key",
          "short_id": "short_id"
        }
      },
      "multiplex": {
        "enabled": false,
        "padding": false
      }
    },
    {
      "type": "direct",
      "tag": "direct-out"
    },
    {
      "type": "direct",
      "bind_interface": "br-lan",
      "tag": "lan-out"
    },
    {
      "type": "dns",
      "tag": "dns-out"
    },
    {
      "type": "block",
      "tag": "block-out"
    }
  ],
  "route": {
    "rules": [
      {
        "type": "logical",
        "mode": "or",
        "rules": [
          {
            "inbound": "dns-in"
          },
          {
            "protocol": "dns"
          }
        ],
        "outbound": "dns-out"
      },
      {
        "geoip": [
          "private"
        ],
        "outbound": "lan-out"
      },
      {
        "domain": [
          "whatismyip.akamai.com",
        ],
        "outbound": "direct-out"
      },
      {
        "type": "logical",
        "mode": "and",
        "rules": [
          {
            "geosite": "geolocation-!cn",
            "invert": true
          },
          {
            "geosite": [
              "cn",
              "category-companies@cn"
            ],
            "geoip": "cn"
          }
        ],
        "outbound": "direct-out"
      },
      {
        "protocol": "quic",
        "outbound": "block-out"
      }
    ],
    "final": "proxy-out",
    "auto_detect_interface": false,
    "geoip": {
      "download_url": "http://192.168.31.108:8080/download/sbox/geoip.db",
      "download_detour": "lan-out"
    },
    "geosite": {
      "download_url": "http://192.168.31.108:8080/download/sbox/geosite.db",
      "download_detour": "lan-out"
    }
  }
}
firedes commented 7 months ago

I've tested this on my pc(Archlinux), same issue. So maybe it is extremely easy to reproduce.

To be clarify, I'd closed all websites and applications relevant to these connections over 30 minutes at least before taking this screenshot. image

UPDATE 2 hours after the previous screenshot, memory usage has come to 90m from the previous 70m. image

{
  "log": {
    "disabled": false,
    "level": "debug",
    "output": "box.log",
    "timestamp": true
  },
  "experimental": {
    "clash_api": {
      "external_controller": "192.168.31.47:9999",
      "external_ui": "yacd",
      "external_ui_download_url": "http://192.168.31.108:8080/download/sbox/gh-pages.zip",
      "external_ui_download_detour": "direct-out",
      "default_mode": "Rule"
    }
  },
  "dns": {
    "servers": [
      {
        "tag": "google",
        "address": "tls://8.8.8.8",
        "detour": "proxy-out"
      },
      {
        "tag": "local",
        "address": "223.5.5.5",
        "detour": "direct-out"
      }
    ],
    "rules": [
      {
        "type": "logical",
        "mode": "and",
        "rules": [
          {
            "geosite": "geolocation-!cn",
            "invert": true
          },
          {
            "geosite": [
              "cn",
              "category-companies@cn"
            ]
          }
        ],
        "server": "local"
      }
    ],
    "strategy": "ipv4_only"
  },
  "inbounds": [
    {
      "type": "tun",
      "tag": "tun-in",
      "interface_name": "tun0",
      "inet4_address": "172.19.0.1/30",
      "mtu": 9000,
      "auto_route": true,
      "strict_route": true,
      "stack": "system",
      "sniff": true,
      "sniff_override_destination": false,
      "sniff_timeout": "300ms",
      "domain_strategy": "prefer_ipv4"
    }
  ],
  "outbounds": [
    {
      "type": "vless",
      "tag": "proxy-out",
      "server": "server",
      "server_port": 443,
      "uuid": "uuid",
      "flow": "xtls-rprx-vision",
      "tls": {
        "enabled": true,
        "disable_sni": false,
        "server_name": "server_name",
        "insecure": false,
        "utls": {
          "enabled": true,
          "fingerprint": "chrome"
        },
        "reality": {
          "enabled": true,
          "public_key": "public_key",
          "short_id": "short_id"
        }
      },
      "multiplex": {
        "enabled": false,
        "padding": false
      }
    },
    {
      "type": "direct",
      "tag": "direct-out"
    },
    {
      "type": "dns",
      "tag": "dns-out"
    }
  ],
  "route": {
    "rules": [
      {
        "protocol": "dns",
        "outbound": "dns-out"
      },
      {
        "geoip": [
          "private"
        ],
        "outbound": "direct-out"
      },
      {
        "port": [
          28287
        ],
        "outbound": "direct-out"
      },
      {
        "domain": [
          "whatismyip.akamai.com",
        ],
        "outbound": "direct-out"
      },
      {
        "type": "logical",
        "mode": "and",
        "rules": [
          {
            "geosite": "geolocation-!cn",
            "invert": true
          },
          {
            "geosite": [
              "cn",
              "category-companies@cn"
            ],
            "geoip": "cn"
          }
        ],
        "outbound": "direct-out"
      }
    ],
    "auto_detect_interface": true,
    "geoip": {
      "download_url": "http://192.168.31.108:8080/download/sbox/geoip.db",
      "download_detour": "direct-out"
    },
    "geosite": {
      "download_url": "http://192.168.31.108:8080/download/sbox/geosite.db",
      "download_detour": "direct-out"
    }
  }
}
SasukeFreestyle commented 7 months ago

I also have this problem. Memory increases over time to the point that it uses over 6GB of RAM, sing-box process then uses over 50% CPU then process freezes and/or system becomes unusable.

Tested with sing-box version 1.6.0 > 1.6.7, 1.7.5, 1.8.0b6.

sudo ss -tulpn | grep <sing-box PID>

image

A quick fix i did was to add this to systemd, once sing-box reaches 4500MB of RAM, its auto-restarted.

Restart=always
MemoryMax=4500M
MemoryLimit=4500M

Ubuntu 22.04.3 LTS

config file.

{
   "log":{
      "level":"info",
      "timestamp":true
   },
   "inbounds":[
      {
         "type":"hysteria2",
         "tag":"hy2-in-80",
         "listen":"::",
         "listen_port":80,
         "sniff":true,
         "sniff_override_destination":true,
         "domain_strategy":"prefer_ipv4",
         "up_mbps":0,
         "down_mbps":0,
         "obfs":{
            "type":"salamander",
            "password":"X"
         },
         "users":[
            {
               "name":"user",
               "password":"X"
            }
         ],
         "ignore_client_bandwidth":true,
         "tls":{
            "enabled":true,
            "certificate_path":"X",
            "key_path":"X"
         }
      },
            {
         "type":"hysteria2",
         "tag":"hy2-in-443",
         "listen":"::",
         "listen_port":80,
         "sniff":true,
         "sniff_override_destination":true,
         "domain_strategy":"prefer_ipv4",
         "up_mbps":0,
         "down_mbps":0,
         "obfs":{
            "type":"salamander",
            "password":"X"
         },
         "users":[
            {
               "name":"user",
               "password":"X"
            }
         ],
         "ignore_client_bandwidth":true,
         "tls":{
            "enabled":true,
            "certificate_path":"X",
            "key_path":"X"
         }
      }
   ],
   "outbounds":[
      {
         "type":"direct",
         "tag":"direct"
      },
      {
         "type":"block",
         "tag":"block"
      },
      {
         "type":"dns",
         "tag":"dns-out"
      }
   ],
   "dns":{
      "disable_cache":true,
      "servers":[
         {
            "tag":"dns-out",
            "address":"udp://127.0.0.53",
            "address_strategy":"prefer_ipv4",
            "strategy":"prefer_ipv4",
            "detour":"direct"
         }
      ]
   },
   "route":{
      "rules":[
         {
            "ip_is_private":true,
            "outbound":"block"
         },
         {
            "rule_set":"geoip-cn",
            "outbound":"block"
         },
         {
            "rule_set":"geoip-ir",
            "outbound":"block"
         },
         {
            "rule_set":"geosite-ir",
            "outbound":"block"
         },
         {
            "protocol":"dns",
            "outbound":"dns-out"
         }
      ],
      "rule_set":[
         {
            "tag":"geoip-cn",
            "type":"local",
            "format":"binary",
            "path":"X"
         },
         {
            "tag":"geoip-ir",
            "type":"local",
            "format":"binary",
            "path":"X"
         },
         {
            "tag":"geosite-ir",
            "type":"local",
            "format":"binary",
            "path":"X"
         }
      ],
      "auto_detect_interface":true
   }
}

EDIT: Since I made this post, RAM usage for sing-box has increased from 1GB to 2GB, with now over 15000 "stale" connections. image

EDIT2: Now over 3GB usage and 22000 connections stale.

EDIT3: Now it crashed. journalctl log below.

Dec 18 00:14:35  sing-box[17676]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 18 00:14:35  sing-box[17676]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7f9dfc]
Dec 18 00:14:35  sing-box[17676]: goroutine 3042 [running]:
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/buf.(*Buffer).Len(...)
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/buf/buffer.go:327
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/bufio.copyPacketWaitWithPool({0x7f006bc031a8?, 0xc000060758}, {0x7f006bc03230, 0xc001175a70}, {0x1610ca8, 0xc00140eb80}, {0x0, 0x0, 0xc86440?}, {0x0, ...}, ...)
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy_direct.go:72 +0x15c
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f006bc03188?, 0xc000799e00?}, {0x7f006bc031a8, 0xc000060758})
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy.go:240 +0x405
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/bufio.CopyPacketConnContextList.func2({0xc00016dd90?, 0x162ad90?})
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy.go:331 +0x66
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/task/task.go:84 +0xce
Dec 18 00:14:35  sing-box[17676]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList in goroutine 2992
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/task/task.go:75 +0x185
Dec 18 00:14:35  sing-box[17676]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 18 00:14:35  sing-box[17676]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7f9dfc]
Dec 18 00:14:35  sing-box[17676]: goroutine 3016 [running]:
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/buf.(*Buffer).Len(...)
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/buf/buffer.go:327
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/bufio.copyPacketWaitWithPool({0x7f006bc031a8?, 0xc000132348}, {0x7f006bc03230, 0xc0011b9710}, {0x1610ca8, 0xc0001a7e80}, {0x0, 0x0, 0xc00029b698?}, {0x0, ...}, ...)
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy_direct.go:72 +0x15c
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f006bc03188?, 0xc001112ff0?}, {0x7f006bc031a8, 0xc000132348})
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy.go:240 +0x405
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/bufio.CopyPacketConnContextList.func2({0xc00100aea0?, 0xc001112e60?})
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy.go:331 +0x66
Dec 18 00:14:35  sing-box[17676]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/task/task.go:84 +0xce
Dec 18 00:14:35  sing-box[17676]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList in goroutine 3014
Dec 18 00:14:35  sing-box[17676]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/task/task.go:75 +0x185
------------------------------------------------------------
Dec 18 00:32:46  sing-box[318]: panic: runtime error: invalid memory address or nil pointer dereference
Dec 18 00:32:46  sing-box[318]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x7f9dfc]
Dec 18 00:32:46  sing-box[318]: goroutine 3559 [running]:
Dec 18 00:32:46  sing-box[318]: github.com/sagernet/sing/common/buf.(*Buffer).Len(...)
Dec 18 00:32:46  sing-box[318]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/buf/buffer.go:327
Dec 18 00:32:46  sing-box[318]: github.com/sagernet/sing/common/bufio.copyPacketWaitWithPool({0x7fd306d7fd18?, 0xc0005e1040}, {0x7fd306d84038, 0xc002695290}, {0x1610ca8, 0xc0028c2680}, {0x0, 0x0, 0x2?}, {0x0, ...}, ...)
Dec 18 00:32:46  sing-box[318]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy_direct.go:72 +0x15c
Dec 18 00:32:46  sing-box[318]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7fd306d83ff8?, 0xc002a987d0?}, {0x7fd306d7fd18, 0xc0005e1040})
Dec 18 00:32:46  sing-box[318]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy.go:240 +0x405
Dec 18 00:32:46  sing-box[318]: github.com/sagernet/sing/common/bufio.CopyPacketConnContextList.func2({0xc0006aa7d0?, 0x530c30?})
Dec 18 00:32:46  sing-box[318]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/bufio/copy.go:331 +0x66
Dec 18 00:32:46  sing-box[318]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
Dec 18 00:32:46  sing-box[318]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/task/task.go:84 +0xce
Dec 18 00:32:46  sing-box[318]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList in goroutine 3557
Dec 18 00:32:46  sing-box[318]:         pkg/mod/github.com/sagernet/sing@v0.3.0-beta.5/common/task/task.go:75 +0x185
lux5am commented 7 months ago

Uses grpc and the memory consumption increased very fast. Just 5 minutes, downloaded 500mb the memory consumption already 200mb. Plain ws is much lower but still raise overtime without going down.

nekohasekai commented 7 months ago

Since you didn't provide a process for reproducing the issue locally, I can assume you are an end user who lacks the knowledge. Since you are reporting on an issue that most people haven't reported, and your issue area has become a gathering place for spam feedback without information, I will close it. I would suggest you use a different protocol or switch to a different project.