xray core stoped randomly when sniffing is enabled

sssagsag commented 5 days ago

Integrity requirements

[X] I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
[X] I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
[X] I searched issues and did not find any similar issues.
[X] The problem can be successfully reproduced in the latest Release

Description

There is a big problem in versions 24.9.30, 24.9.19, 24, 9.16 (version 1.8.24 and 24.9.7 are not a problem), when the tls sniffing feature is active on a high-traffic server with about 100 online users with 5000 open connections, the xray core crashes randomly for 5 to 10 minutes and the xray must be run again (my server resources It is good and there is no shortage (8 cores with 16 gigs of RAM), I checked this problem on a low traffic server and it takes hours for the core to stop working. Also, all my users use mux The problem does not occur when Sniff is turned off and the core is active for days.

Client config


{
  "log": {
    "access": "",
    "error": "",
    "loglevel": "warning"
  },
  "dns": {
    "servers": [
      "1.1.1.1",
      "8.8.8.8"
    ],
    "queryStrategy": "UseIPv4"
  },
  "inbounds": [
    {
      "tag": "socks",
      "port": 10808,
      "listen": "127.0.0.1",
      "protocol": "socks",
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls"
        ],
        "routeOnly": false
      },
      "settings": {
        "auth": "noauth",
        "udp": true,
        "allowTransparent": false
      }
    },
    {
      "tag": "http",
      "port": 10809,
      "listen": "127.0.0.1",
      "protocol": "http",
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls"
        ],
        "routeOnly": false
      },
      "settings": {
        "auth": "noauth",
        "udp": true,
        "allowTransparent": false
      }
    }
  ],
  "outbounds": [
    {
      "tag": "proxy",
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "",
            "port": 443,
            "users": [
              {
                "id": "",
                "alterId": 0,
                "email": "t@t.tt",
                "security": "auto",
                "encryption": "none"
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "tlsSettings": {
          "allowInsecure": true,
          "fingerprint": "chrome",
          "serverName": ""
        },
        "wsSettings": {
          "headers": {
            "Host": ""          },
          "path": "/?ed=2560"
        },
        "sockopt": {
          "dialerProxy": "fragment0",
          "tcpKeepAliveIdle": 100,
          "mark": 255,
          "tcpNoDelay": true
        }
      },
      "mux": {
        "concurrency": 8,
        "enabled": true,
        "xudpConcurrency": 8,
        "xudpProxyUDP443": "reject"
      }
    },
    {
      "tag": "fragment0",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs",
        "fragment": {
          "packets": "1-1",
          "length": "1-10",
          "interval": "1-10"
        }
      },
      "streamSettings": {
        "sockopt": {
          "tcpNoDelay": true,
          "tcpKeepAliveIdle": 100
        }
      }
    }
  ],
  "routing": {
    "domainStrategy": "IPIfNonMatch"
  },
  "remarks": ""
}

Server config


{
  "inbounds": [
    {
      "listen": null,
      "port": 8080,
      "protocol": "vless",
      "settings": {
        "clients": [
        ],
        "decryption": "none",
        "fallbacks": []
      },
      "sniffing": {
        "destOverride": [
          "tls"
        ],
        "enabled": true,
        "metadataOnly": false,
        "routeOnly": false
      },
      "streamSettings": {
        "network": "ws",
        "security": "none",
        "wsSettings": {
          "acceptProxyProtocol": false,
          "headers": {},
          "host": "",
          "path": "/"
        }
      },
      "tag": "inbound-8080"
    }
  ],
  "log": {
    "access": "none",
    "error": "/root/error.log",
    "loglevel": "debug"
  },
  "observatory": null,
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "ForceIPv4"
      },
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    }
     ],
  "policy": {
    "levels": {
      "0": {
        "statsUserDownlink": true,
        "statsUserUplink": true
      }
    },
    "system": {
      "statsInboundDownlink": true,
      "statsInboundUplink": true
    }
  },
  "reverse": null,
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "inboundTag": [
          "api"
        ],
        "outboundTag": "api",
        "type": "field"
      },
      {
        "outboundTag": "blocked",
        "protocol": [
          "bittorrent"
        ],
        "type": "field"
      }
    ]
  },
  "stats": {},
  "transport": null
}

Server log

serverroot.txt xray-debug2.log

sssagsag commented 5 days ago

please check this problem @mmmray @yuhan6665 @RPRX

Fangliding commented 5 days ago

dup https://github.com/XTLS/Xray-core/issues/3904

sssagsag commented 5 days ago

dup #3904

What exactly is the connection between #3904 and my problem?

Fangliding commented 5 days ago

dup #3904

What exactly is the connection between #3904 and my problem?

See the pull request comment(if you understand)

MHSanaei commented 5 days ago

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

sssagsag commented 5 days ago

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

Yes, I will definitely test it now

sssagsag commented 5 days ago

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

The problem is not solved, the core is stopped one after another

sssagsag commented 5 days ago

dup #3904

What exactly is the connection between #3904 and my problem?

See the pull request comment(if you understand)

The problem is not solved, the core is stopped one after another

Fangliding commented 5 days ago

I need to see new log file

sssagsag commented 5 days ago

I need to see new log file

OK, I will capture it and send it to you

sssagsag commented 5 days ago

I need to see new log file

newxray.txt xray-debugnew.log

Fangliding commented 5 days ago

I understand, I'll handle it tomorrow

sssagsag commented 5 days ago

I understand, I'll handle it tomorrow

thank you sir

Fangliding commented 5 days ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

sssagsag commented 5 days ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914 Is this version that you builded different from the new 24.10.16?

Fangliding commented 5 days ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914 Is this version that you builded different from the new 24.10.16?

It's a test version, not on the main branch, not included in 24.10.16

Fangliding commented 5 days ago

log

sssagsag commented 4 days ago

log

I put your modified your xray core on one super high-traffic server and it has been online for about 5 hour now, and apparently the problem has been completely solved.

sssagsag commented 4 days ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

10 hours online without any problem with the worst route and the worst possible settings and the large number of online users, the problem is completely solved, thank you for your attention , please add to main branch

RPRX commented 4 days ago

@Fangliding 开个 pr

sssagsag commented 4 days ago

19 hours online without any problem

IMG_20241017_095517

sssagsag commented 3 days ago

update IMG_20241018_133234 2 Days online without any problem

realartin commented 3 days ago

i guess i have a similar issue :

xray core v24.9.30

github.com/xtls/xray-core/common/task/task.go:12 +0x22 github.com/xtls/xray-core/common/task.Run.func1(0xc0066067e0?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 23562 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 24996 [select]: github.com/xtls/xray-core/transport/internet/splithttp.(*requestHandler).ServeHTTP(0xc00118eb40, {0x153af30, 0xc005fa3d88}, 0xc005736140) github.com/xtls/xray-core/transport/internet/splithttp/hub.go:217 +0xdea golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x152d320?, 0xc00118eb40?}, 0xc001caf140?}, {0x153af30, 0xc005fa3d88}, 0xc005736140) golang.org/x/net@v0.29.0/http2/h2c/h2c.go:125 +0x673 net/http.serverHandler.ServeHTTP({0x1ff?}, {0x153af30?, 0xc005fa3d88?}, 0xc000492000?) net/http/server.go:3210 +0x8e net/http.initALPNRequest.ServeHTTP({{0x153d300?, 0xc0032aeb40?}, 0xc000122e08?, {0xc000134288?}}, {0x153af30, 0xc005fa3d88}, 0xc005736140) net/http/server.go:3819 +0x231 net/http.(*http2serverConn).runHandler(0x445edb?, 0x0?, 0x0?, 0x0?) net/http/h2_bundle.go:6249 +0xf5 created by net/http.(*http2serverConn).scheduleHandler in goroutine 23156 net/http/h2_bundle.go:6183 +0x21d goroutine 21880 [select, 2 minutes]: github.com/xtls/xray-core/common/task.Run({0x153d300, 0xc006155d70}, {0xc005a9bec8, 0x2, 0xc006e79bb0?}) github.com/xtls/xray-core/common/task/task.go:48 +0x265 github.com/xtls/xray-core/proxy/vmess/inbound.(*Handler).Process(0xc00032fa40, {0x153d300, 0xc0062bd3b0}, 0x151dbd0?, {0x1544698, 0xc00015d8a0}, {0x153f720, 0xc000045490}) github.com/xtls/xray-core/proxy/vmess/inbound/inbound.go:298 +0xdef github.com/xtls/xray-core/app/proxyman/inbound.(*tcpWorker).callback(0xc000e8b540, {0x1544698, 0xc00015d8a0}) github.com/xtls/xray-core/app/proxyman/inbound/worker.go:109 +0x5e7 created by github.com/xtls/xray-core/app/proxyman/inbound.(*tcpWorker).Start.func1 in goroutine 11 github.com/xtls/xray-core/app/proxyman/inbound/worker.go:123 +0x7a goroutine 25381 [select]: github.com/xtls/xray-core/transport/pipe.(*pipe).ReadMultiBuffer(0xc004b71b30) github.com/xtls/xray-core/transport/pipe/impl.go:100 +0xeb github.com/xtls/xray-core/transport/pipe.(*Reader).ReadMultiBuffer(0xc004b2a540?) github.com/xtls/xray-core/transport/pipe/reader.go:16 +0x16 github.com/xtls/xray-core/common/buf.copyInternal({0x152b600, 0xc0071af2d8}, {0x152d9c0, 0xc004b2a540}, 0xc00711d848) github.com/xtls/xray-core/common/buf/copy.go:93 +0x5b github.com/xtls/xray-core/common/buf.Copy({0x152b600, 0xc0071af2d8}, {0x152d9c0, 0xc004b2a540}, {0xc00423df58, 0x1, 0xc004b2a4e0?}) github.com/xtls/xray-core/common/buf/copy.go:116 +0x98 github.com/xtls/xray-core/proxy/vless/inbound.(*Handler).Process.func5() github.com/xtls/xray-core/proxy/vless/inbound/inbound.go:567 +0x3ff github.com/xtls/xray-core/common/task.Run.func1(0xc006553f50?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 25332 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 17308 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7c72e93d27d8, 0x72) runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc004d1a900?, 0xc001efe000?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc004d1a900, {0xc001efe000, 0x2000, 0x2000}) internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc004d1a900, {0xc001efe000?, 0x410845?, 0x30?}) net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc002625928, {0xc001efe000?, 0x2000?, 0xc001efe000?}) net/net.go:189 +0x45 github.com/xtls/xray-core/common/buf.(*Buffer).ReadFrom(...) github.com/xtls/xray-core/common/buf/buffer.go:295 github.com/xtls/xray-core/common/buf.ReadBuffer({0x152d820, 0xc002625928}) github.com/xtls/xray-core/common/buf/reader.go:30 +0x66 github.com/xtls/xray-core/common/buf.(*ReadVReader).ReadMultiBuffer(0xc0066b3bd0) github.com/xtls/xray-core/common/buf/readv_reader.go:125 +0x65 github.com/xtls/xray-core/common/buf.copyInternal({0x152da40, 0xc0066b3bd0}, {0x152b660, 0xc001473dc0}, 0xc006d61f20) github.com/xtls/xray-core/common/buf/copy.go:93 +0x5b github.com/xtls/xray-core/common/buf.Copy({0x152da40, 0xc0066b3bd0}, {0x152b660, 0xc001473dc0}, {0xc004cb2f30, 0x1, 0x10c3d80?}) github.com/xtls/xray-core/common/buf/copy.go:116 +0x98 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process.func3() github.com/xtls/xray-core/proxy/wireguard/client.go:206 +0xf9 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process.OnSuccess.func7() github.com/xtls/xray-core/common/task/task.go:12 +0x22 github.com/xtls/xray-core/common/task.Run.func1(0x24?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 17301 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 17295 [select, 4 minutes]: github.com/xtls/xray-core/transport/internet/splithttp.(*requestHandler).ServeHTTP(0xc00118eb40, {0x153af30, 0xc0026257c8}, 0xc0065b9900) github.com/xtls/xray-core/transport/internet/splithttp/hub.go:217 +0xdea golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x152d320?, 0xc00118eb40?}, 0xc001caf140?}, {0x153af30, 0xc0026257c8}, 0xc0065b9900) golang.org/x/net@v0.29.0/http2/h2c/h2c.go:125 +0x673 net/http.serverHandler.ServeHTTP({0x0?}, {0x153af30?, 0xc0026257c8?}, 0xc0011fa300?) net/http/server.go:3210 +0x8e net/http.initALPNRequest.ServeHTTP({{0x153d300?, 0xc004ce9380?}, 0xc000004708?, {0xc000134288?}}, {0x153af30, 0xc0026257c8}, 0xc0065b9900) net/http/server.go:3819 +0x231 net/http.(*http2serverConn).runHandler(0xaad7e8?, 0xc000e8b540?, 0x1544698?, 0xc000284b00?) net/http/h2_bundle.go:6249 +0xf5 created by net/http.(*http2serverConn).scheduleHandler in goroutine 12548 net/http/h2_bundle.go:6183 +0x21d goroutine 19879 [select, 2 minutes]: github.com/xtls/xray-core/common/task.Run({0x153d338, 0xc006655900}, {0xc003e87cd0, 0x2, 0xc000199480?}) github.com/xtls/xray-core/common/task/task.go:48 +0x265 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process(0xc0001dcee0, {0x153d300, 0xc003fe48a0}, 0xc00695edc0, {0x153a640, 0xc000199480}) github.com/xtls/xray-core/proxy/wireguard/client.go:230 +0xc85 github.com/xtls/xray-core/app/proxyman/outbound.(*Handler).Dispatch(0xc000199480, {0x153d300, 0xc003fe48a0}, 0xc00695edc0) github.com/xtls/xray-core/app/proxyman/outbound/handler.go:211 +0x388 github.com/xtls/xray-core/app/dispatcher.(*DefaultDispatcher).routedDispatch(0xc0010e6c00, {0x153d300, 0xc003fe48a0}, 0xc00695edc0, {{0x153d450, 0xc00321458c}, 0x35, 0x3})

XTLS / Xray-core