xray core stoped randomly when sniffing is enabled

sssagsag commented 1 month ago

Integrity requirements

[X] I confirm that I have read the documentation, understand the meaning of all the configuration items I wrote, and did not pile up seemingly useful options or default values.
[X] I provided the complete config and logs, rather than just providing the truncated parts based on my own judgment.
[X] I searched issues and did not find any similar issues.
[X] The problem can be successfully reproduced in the latest Release

Description

There is a big problem in versions 24.9.30, 24.9.19, 24, 9.16 (version 1.8.24 and 24.9.7 are not a problem), when the tls sniffing feature is active on a high-traffic server with about 100 online users with 5000 open connections, the xray core crashes randomly for 5 to 10 minutes and the xray must be run again (my server resources It is good and there is no shortage (8 cores with 16 gigs of RAM), I checked this problem on a low traffic server and it takes hours for the core to stop working. Also, all my users use mux The problem does not occur when Sniff is turned off and the core is active for days.

Client config


{
  "log": {
    "access": "",
    "error": "",
    "loglevel": "warning"
  },
  "dns": {
    "servers": [
      "1.1.1.1",
      "8.8.8.8"
    ],
    "queryStrategy": "UseIPv4"
  },
  "inbounds": [
    {
      "tag": "socks",
      "port": 10808,
      "listen": "127.0.0.1",
      "protocol": "socks",
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls"
        ],
        "routeOnly": false
      },
      "settings": {
        "auth": "noauth",
        "udp": true,
        "allowTransparent": false
      }
    },
    {
      "tag": "http",
      "port": 10809,
      "listen": "127.0.0.1",
      "protocol": "http",
      "sniffing": {
        "enabled": true,
        "destOverride": [
          "http",
          "tls"
        ],
        "routeOnly": false
      },
      "settings": {
        "auth": "noauth",
        "udp": true,
        "allowTransparent": false
      }
    }
  ],
  "outbounds": [
    {
      "tag": "proxy",
      "protocol": "vless",
      "settings": {
        "vnext": [
          {
            "address": "",
            "port": 443,
            "users": [
              {
                "id": "",
                "alterId": 0,
                "email": "t@t.tt",
                "security": "auto",
                "encryption": "none"
              }
            ]
          }
        ]
      },
      "streamSettings": {
        "network": "ws",
        "security": "tls",
        "tlsSettings": {
          "allowInsecure": true,
          "fingerprint": "chrome",
          "serverName": ""
        },
        "wsSettings": {
          "headers": {
            "Host": ""          },
          "path": "/?ed=2560"
        },
        "sockopt": {
          "dialerProxy": "fragment0",
          "tcpKeepAliveIdle": 100,
          "mark": 255,
          "tcpNoDelay": true
        }
      },
      "mux": {
        "concurrency": 8,
        "enabled": true,
        "xudpConcurrency": 8,
        "xudpProxyUDP443": "reject"
      }
    },
    {
      "tag": "fragment0",
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "AsIs",
        "fragment": {
          "packets": "1-1",
          "length": "1-10",
          "interval": "1-10"
        }
      },
      "streamSettings": {
        "sockopt": {
          "tcpNoDelay": true,
          "tcpKeepAliveIdle": 100
        }
      }
    }
  ],
  "routing": {
    "domainStrategy": "IPIfNonMatch"
  },
  "remarks": ""
}

Server config


{
  "inbounds": [
    {
      "listen": null,
      "port": 8080,
      "protocol": "vless",
      "settings": {
        "clients": [
        ],
        "decryption": "none",
        "fallbacks": []
      },
      "sniffing": {
        "destOverride": [
          "tls"
        ],
        "enabled": true,
        "metadataOnly": false,
        "routeOnly": false
      },
      "streamSettings": {
        "network": "ws",
        "security": "none",
        "wsSettings": {
          "acceptProxyProtocol": false,
          "headers": {},
          "host": "",
          "path": "/"
        }
      },
      "tag": "inbound-8080"
    }
  ],
  "log": {
    "access": "none",
    "error": "/root/error.log",
    "loglevel": "debug"
  },
  "observatory": null,
  "outbounds": [
    {
      "protocol": "freedom",
      "settings": {
        "domainStrategy": "ForceIPv4"
      },
      "tag": "direct"
    },
    {
      "protocol": "blackhole",
      "settings": {},
      "tag": "blocked"
    }
     ],
  "policy": {
    "levels": {
      "0": {
        "statsUserDownlink": true,
        "statsUserUplink": true
      }
    },
    "system": {
      "statsInboundDownlink": true,
      "statsInboundUplink": true
    }
  },
  "reverse": null,
  "routing": {
    "domainStrategy": "IPIfNonMatch",
    "rules": [
      {
        "inboundTag": [
          "api"
        ],
        "outboundTag": "api",
        "type": "field"
      },
      {
        "outboundTag": "blocked",
        "protocol": [
          "bittorrent"
        ],
        "type": "field"
      }
    ]
  },
  "stats": {},
  "transport": null
}

Server log

serverroot.txt xray-debug2.log

sssagsag commented 1 month ago

please check this problem @mmmray @yuhan6665 @RPRX

Fangliding commented 1 month ago

dup https://github.com/XTLS/Xray-core/issues/3904

sssagsag commented 1 month ago

dup #3904

What exactly is the connection between #3904 and my problem?

Fangliding commented 1 month ago

dup #3904

What exactly is the connection between #3904 and my problem?

See the pull request comment(if you understand)

MHSanaei commented 1 month ago

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

sssagsag commented 1 month ago

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

Yes, I will definitely test it now

sssagsag commented 1 month ago

@sssagsag Bro, just download that PR build and test it. https://github.com/XTLS/Xray-core/actions/runs/11339237462

The problem is not solved, the core is stopped one after another

sssagsag commented 1 month ago

dup #3904

What exactly is the connection between #3904 and my problem?

See the pull request comment(if you understand)

The problem is not solved, the core is stopped one after another

Fangliding commented 1 month ago

I need to see new log file

sssagsag commented 1 month ago

I need to see new log file

OK, I will capture it and send it to you

sssagsag commented 1 month ago

I need to see new log file

newxray.txt xray-debugnew.log

Fangliding commented 1 month ago

I understand, I'll handle it tomorrow

sssagsag commented 1 month ago

I understand, I'll handle it tomorrow

thank you sir

Fangliding commented 1 month ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

sssagsag commented 1 month ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914 Is this version that you builded different from the new 24.10.16?

Fangliding commented 1 month ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914 Is this version that you builded different from the new 24.10.16?

It's a test version, not on the main branch, not included in 24.10.16

Fangliding commented 1 month ago

log

sssagsag commented 1 month ago

log

I put your modified your xray core on one super high-traffic server and it has been online for about 5 hour now, and apparently the problem has been completely solved.

sssagsag commented 1 month ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

10 hours online without any problem with the worst route and the worst possible settings and the large number of online users, the problem is completely solved, thank you for your attention , please add to main branch

RPRX commented 1 month ago

@Fangliding 开个 pr

realartin commented 1 month ago

i guess i have a similar issue :

xray core v24.9.30

github.com/xtls/xray-core/common/task/task.go:12 +0x22 github.com/xtls/xray-core/common/task.Run.func1(0xc0066067e0?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 23562 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 24996 [select]: github.com/xtls/xray-core/transport/internet/splithttp.(*requestHandler).ServeHTTP(0xc00118eb40, {0x153af30, 0xc005fa3d88}, 0xc005736140) github.com/xtls/xray-core/transport/internet/splithttp/hub.go:217 +0xdea golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x152d320?, 0xc00118eb40?}, 0xc001caf140?}, {0x153af30, 0xc005fa3d88}, 0xc005736140) golang.org/x/net@v0.29.0/http2/h2c/h2c.go:125 +0x673 net/http.serverHandler.ServeHTTP({0x1ff?}, {0x153af30?, 0xc005fa3d88?}, 0xc000492000?) net/http/server.go:3210 +0x8e net/http.initALPNRequest.ServeHTTP({{0x153d300?, 0xc0032aeb40?}, 0xc000122e08?, {0xc000134288?}}, {0x153af30, 0xc005fa3d88}, 0xc005736140) net/http/server.go:3819 +0x231 net/http.(*http2serverConn).runHandler(0x445edb?, 0x0?, 0x0?, 0x0?) net/http/h2_bundle.go:6249 +0xf5 created by net/http.(*http2serverConn).scheduleHandler in goroutine 23156 net/http/h2_bundle.go:6183 +0x21d goroutine 21880 [select, 2 minutes]: github.com/xtls/xray-core/common/task.Run({0x153d300, 0xc006155d70}, {0xc005a9bec8, 0x2, 0xc006e79bb0?}) github.com/xtls/xray-core/common/task/task.go:48 +0x265 github.com/xtls/xray-core/proxy/vmess/inbound.(*Handler).Process(0xc00032fa40, {0x153d300, 0xc0062bd3b0}, 0x151dbd0?, {0x1544698, 0xc00015d8a0}, {0x153f720, 0xc000045490}) github.com/xtls/xray-core/proxy/vmess/inbound/inbound.go:298 +0xdef github.com/xtls/xray-core/app/proxyman/inbound.(*tcpWorker).callback(0xc000e8b540, {0x1544698, 0xc00015d8a0}) github.com/xtls/xray-core/app/proxyman/inbound/worker.go:109 +0x5e7 created by github.com/xtls/xray-core/app/proxyman/inbound.(*tcpWorker).Start.func1 in goroutine 11 github.com/xtls/xray-core/app/proxyman/inbound/worker.go:123 +0x7a goroutine 25381 [select]: github.com/xtls/xray-core/transport/pipe.(*pipe).ReadMultiBuffer(0xc004b71b30) github.com/xtls/xray-core/transport/pipe/impl.go:100 +0xeb github.com/xtls/xray-core/transport/pipe.(*Reader).ReadMultiBuffer(0xc004b2a540?) github.com/xtls/xray-core/transport/pipe/reader.go:16 +0x16 github.com/xtls/xray-core/common/buf.copyInternal({0x152b600, 0xc0071af2d8}, {0x152d9c0, 0xc004b2a540}, 0xc00711d848) github.com/xtls/xray-core/common/buf/copy.go:93 +0x5b github.com/xtls/xray-core/common/buf.Copy({0x152b600, 0xc0071af2d8}, {0x152d9c0, 0xc004b2a540}, {0xc00423df58, 0x1, 0xc004b2a4e0?}) github.com/xtls/xray-core/common/buf/copy.go:116 +0x98 github.com/xtls/xray-core/proxy/vless/inbound.(*Handler).Process.func5() github.com/xtls/xray-core/proxy/vless/inbound/inbound.go:567 +0x3ff github.com/xtls/xray-core/common/task.Run.func1(0xc006553f50?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 25332 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 17308 [IO wait, 2 minutes]: internal/poll.runtime_pollWait(0x7c72e93d27d8, 0x72) runtime/netpoll.go:351 +0x85 internal/poll.(*pollDesc).wait(0xc004d1a900?, 0xc001efe000?, 0x0) internal/poll/fd_poll_runtime.go:84 +0x27 internal/poll.(*pollDesc).waitRead(...) internal/poll/fd_poll_runtime.go:89 internal/poll.(*FD).Read(0xc004d1a900, {0xc001efe000, 0x2000, 0x2000}) internal/poll/fd_unix.go:165 +0x27a net.(*netFD).Read(0xc004d1a900, {0xc001efe000?, 0x410845?, 0x30?}) net/fd_posix.go:55 +0x25 net.(*conn).Read(0xc002625928, {0xc001efe000?, 0x2000?, 0xc001efe000?}) net/net.go:189 +0x45 github.com/xtls/xray-core/common/buf.(*Buffer).ReadFrom(...) github.com/xtls/xray-core/common/buf/buffer.go:295 github.com/xtls/xray-core/common/buf.ReadBuffer({0x152d820, 0xc002625928}) github.com/xtls/xray-core/common/buf/reader.go:30 +0x66 github.com/xtls/xray-core/common/buf.(*ReadVReader).ReadMultiBuffer(0xc0066b3bd0) github.com/xtls/xray-core/common/buf/readv_reader.go:125 +0x65 github.com/xtls/xray-core/common/buf.copyInternal({0x152da40, 0xc0066b3bd0}, {0x152b660, 0xc001473dc0}, 0xc006d61f20) github.com/xtls/xray-core/common/buf/copy.go:93 +0x5b github.com/xtls/xray-core/common/buf.Copy({0x152da40, 0xc0066b3bd0}, {0x152b660, 0xc001473dc0}, {0xc004cb2f30, 0x1, 0x10c3d80?}) github.com/xtls/xray-core/common/buf/copy.go:116 +0x98 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process.func3() github.com/xtls/xray-core/proxy/wireguard/client.go:206 +0xf9 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process.OnSuccess.func7() github.com/xtls/xray-core/common/task/task.go:12 +0x22 github.com/xtls/xray-core/common/task.Run.func1(0x24?) github.com/xtls/xray-core/common/task/task.go:28 +0x28 created by github.com/xtls/xray-core/common/task.Run in goroutine 17301 github.com/xtls/xray-core/common/task/task.go:27 +0xdb goroutine 17295 [select, 4 minutes]: github.com/xtls/xray-core/transport/internet/splithttp.(*requestHandler).ServeHTTP(0xc00118eb40, {0x153af30, 0xc0026257c8}, 0xc0065b9900) github.com/xtls/xray-core/transport/internet/splithttp/hub.go:217 +0xdea golang.org/x/net/http2/h2c.h2cHandler.ServeHTTP({{0x152d320?, 0xc00118eb40?}, 0xc001caf140?}, {0x153af30, 0xc0026257c8}, 0xc0065b9900) golang.org/x/net@v0.29.0/http2/h2c/h2c.go:125 +0x673 net/http.serverHandler.ServeHTTP({0x0?}, {0x153af30?, 0xc0026257c8?}, 0xc0011fa300?) net/http/server.go:3210 +0x8e net/http.initALPNRequest.ServeHTTP({{0x153d300?, 0xc004ce9380?}, 0xc000004708?, {0xc000134288?}}, {0x153af30, 0xc0026257c8}, 0xc0065b9900) net/http/server.go:3819 +0x231 net/http.(*http2serverConn).runHandler(0xaad7e8?, 0xc000e8b540?, 0x1544698?, 0xc000284b00?) net/http/h2_bundle.go:6249 +0xf5 created by net/http.(*http2serverConn).scheduleHandler in goroutine 12548 net/http/h2_bundle.go:6183 +0x21d goroutine 19879 [select, 2 minutes]: github.com/xtls/xray-core/common/task.Run({0x153d338, 0xc006655900}, {0xc003e87cd0, 0x2, 0xc000199480?}) github.com/xtls/xray-core/common/task/task.go:48 +0x265 github.com/xtls/xray-core/proxy/wireguard.(*Handler).Process(0xc0001dcee0, {0x153d300, 0xc003fe48a0}, 0xc00695edc0, {0x153a640, 0xc000199480}) github.com/xtls/xray-core/proxy/wireguard/client.go:230 +0xc85 github.com/xtls/xray-core/app/proxyman/outbound.(*Handler).Dispatch(0xc000199480, {0x153d300, 0xc003fe48a0}, 0xc00695edc0) github.com/xtls/xray-core/app/proxyman/outbound/handler.go:211 +0x388 github.com/xtls/xray-core/app/dispatcher.(*DefaultDispatcher).routedDispatch(0xc0010e6c00, {0x153d300, 0xc003fe48a0}, 0xc00695edc0, {{0x153d450, 0xc00321458c}, 0x35, 0x3})

sssagsag commented 1 month ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11358760914

Since I put your modified core on the server a few days ago, the kernel has stopped 3 times. Please see the log file. output.txt

Fangliding commented 1 month ago

This log is incomplete and lacks the key information

sssagsag commented 1 month ago

This log is incomplete and lacks the key information

The log is 25 GB in size and I am unable to extract it.

Fangliding commented 1 month ago

Logs can be truncated, but they should be truncated at the end rather than the beginning

sssagsag commented 4 weeks ago

Logs can be truncated, but they should be truncated at the end rather than the beginning

I put the error data at times, but some of them have disappeared, what mode do you suggest to diagnose the problem and make the log smaller? no i use debug mode

realartin commented 3 weeks ago

Does this fixed on latest release ?

sssagsag commented 3 weeks ago

Does this fixed on latest release ?

I put the new version on high traffic servers to see if it is fixed or not

RPRX commented 3 weeks ago

Does this fixed on latest release ?

解决了，但还没根治，这个 issue 需要继续保持 open

realartin commented 3 weeks ago

Does this fixed on latest release ?

解决了，但还没根治，这个 issue 需要继续保持 open

Can we use mux and snifing on or mux is still unstable the core ?

Tnx for your help

sssagsag commented 3 weeks ago

Does this fixed on latest release ?

I put the new version on high traffic servers to see if it is fixed or not

version v24.10.31 stopped after 5 hours with 5 high load server

sssagsag commented 3 weeks ago

after 24 hours xray running core is stoped 2024/11/01 15:45:38 DEBUG - XRAY: github.com/xtls/xray-core/common/protocol/quic.SniffQUIC({0xc001d96000, 0x4b0, 0x2000}) 2024/11/01 15:45:38 DEBUG - XRAY: panic: runtime error: slice bounds out of range [18:17]

ImMohammad20000 commented 3 weeks ago

after 24 hours xray running core is stoped 2024/11/01 15:45:38 DEBUG - XRAY: github.com/xtls/xray-core/common/protocol/quic.SniffQUIC({0xc001d96000, 0x4b0, 0x2000}) 2024/11/01 15:45:38 DEBUG - XRAY: panic: runtime error: slice bounds out of range [18:17]

Do you use quic sniffer? if the answer is yes disable quick sniffer and test again

Fangliding commented 3 weeks ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

sssagsag commented 2 weeks ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

I didn't test your version, I tested the new version 24.11.5 directly.

nohup: ignoring input Xray 24.11.5 (Xray, Penetrates Everything.) afc7ec5 (go1.23.2 linux/amd64) A unified platform for anti-censorship. 2024/11/06 08:03:07 [Info] infra/conf/serial: Reading config: &{Name:/root/config.json Format:json} panic: runtime error: slice bounds out of range [90:89]

goroutine 44557 [running]: github.com/xtls/xray-core/common/protocol/quic.SniffQUIC({0xc001a72000, 0x4e2, 0x2000}) github.com/xtls/xray-core/common/protocol/quic/sniff.go:162 +0x1ed0 github.com/xtls/xray-core/app/dispatcher.NewSniffer.func4({0x80?, 0x13278a0?}, {0xc001a72000?, 0x1?, 0x4e2?}) github.com/xtls/xray-core/app/dispatcher/sniffer.go:41 +0x25 github.com/xtls/xray-core/app/dispatcher.(Sniffer).Sniff(0xc003a2fe78, {0x1549c28, 0xc003befad0}, {0xc001a72000, 0x4e2, 0x2000}, 0x3) github.com/xtls/xray-core/app/dispatcher/sniffer.go:65 +0xcc github.com/xtls/xray-core/app/dispatcher.sniffer.func1({0x1549c28, 0xc003befad0}, 0xc003befb60, 0xc003befb90, 0xc003a2fe78, 0x3) github.com/xtls/xray-core/app/dispatcher/default.go:383 +0x125 github.com/xtls/xray-core/app/dispatcher.sniffer({0x1549c28, 0xc003befad0}, 0xc003befb60, 0x0, 0x3) github.com/xtls/xray-core/app/dispatcher/default.go:393 +0x116 github.com/xtls/xray-core/app/dispatcher.(DefaultDispatcher).Dispatch.func1() github.com/xtls/xray-core/app/dispatcher/default.go:275 +0x10f created by github.com/xtls/xray-core/app/dispatcher.(*DefaultDispatcher).Dispatch in goroutine 44556 github.com/xtls/xray-core/app/dispatcher/default.go:270 +0x47f

sssagsag commented 2 weeks ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

I put this version and will keep you updated on the status.

sssagsag commented 2 weeks ago

Try this version https://github.com/XTLS/Xray-core/actions/runs/11645174316

after 4 hour online your version crash xray core seem sloved will keep you updated on the status.

sssagsag commented 2 weeks ago

update after 10 hours online without crash {B90740C6-5DEE-4985-AA13-AF1126F7668F}

sssagsag commented 1 week ago

problem sloved , thanks to @Fangliding and @RPRX , now update 24.11.11 work perfectly

RPRX commented 1 week ago

这个 issue 需要保持开启，https://github.com/XTLS/Xray-core/pull/3908 https://github.com/XTLS/Xray-core/pull/3921 https://github.com/XTLS/Xray-core/pull/3978 这三个都只是治标，还没有治本。话说 @Fangliding 写一下 https://github.com/XTLS/Xray-core/pull/3867#issuecomment-2412847316

XTLS / Xray-core