search

SagerNet / sing-box

The universal proxy platform
https://sing-box.sagernet.org/
Other
18.57k stars 2.22k forks source link

Panic: repeated read on failed websocket connection #473

Closed malikshi closed 1 year ago

malikshi commented 1 year ago

Welcome

Description of the problem

i saw panic error in log, just check it today. i am using automatic build every hour so the version should keep with latest commit. i am using nginx configured ssl and can be accessed as normal sites and put websocket path forwarded to sing-box.

  1. i setup nginx static sites with port 443, e.g https://github.com/XTLS/Xray-examples/blob/main/VLESS-WSS-Nginx/nginx.conf 

    server {
# dedicated port 80
listen                  80 reuseport;
listen                  [::]:80 reuseport;
listen                  443 ssl http2 reuseport;
listen                  [::]:443 ssl http2 reuseport;
server_name             domain;
root                    /var/www;
# SSL
ssl_certificate         /etc/letsencrypt/live/domain/fullchain.pem;
ssl_certificate_key     /etc/letsencrypt/live/domain/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/domain/chain.pem;

location ^~ /vmess-ws {
if ($http_upgrade != "websocket") {
    return 404;
}
    proxy_pass http://127.0.0.1:2080;
    proxy_set_header Host $host;
proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_read_timeout 52w;
}

  2. i don't put anything weird set here, since i am running 3 server 2 of them stilll using build 20 march and works normally. just last server with recent commit having trouble, i didn't see the panic log /var/log/box.log but i see the panic log in /var/log/syslog

Version of sing-box

```console $ sing-box version sing-box version unknown Environment: go1.20.2 linux/amd64 Tags: with_clash_api,with_grpc,with_wireguard,with_utls,with_gvisor,with_quic,with_dhcp,with_ech,with_reality_server CGO: disabled ```

Server and client configuration file

```console { "log": { "disabled": false, "level": "info", "output": "/var/log/box.log", "timestamp": true }, "dns": { "servers": [ { "tag": "local", "address": "https://1.1.1.1/dns-query", "address_strategy": "ipv4_only", "strategy": "ipv4_only", "detour": "direct" }, { "tag": "block", "address": "rcode://success" } ], "rules": [], "final": "local", "strategy": "", "disable_cache": false, "disable_expire": false }, "experimental": {}, "outbounds": [ { "type": "direct", "tag": "direct" }, { "type": "block", "tag": "block" }, { "type": "dns", "tag": "dns-out" } ], "route": { "final": "direct", "auto_detect_interface": true, "geoip": { "path": "/tmp/geoip.db", "download_url": "https://github.com/malikshi/sing-box-geo/releases/latest/download/geoip.db", "download_detour": "direct" }, "geosite": { "path": "/tmp/geosite.db", "download_url": "https://github.com/malikshi/sing-box-geo/releases/latest/download/geosite.db", "download_detour": "direct" }, "rules": [ { "port": 53, "outbound": "dns-out" } ] }, "inbound": [ { "type":"vmess", "tag":"vmess-ws", "listen":"127.0.0.1", "listen_port":2080, "tcp_fast_open":true, "sniff":true, "sniff_override_destination":true, "sniff_timeout":"600ms", "udp_fragment":false, "udp_timeout":300, "proxy_protocol":false, "proxy_protocol_accept_no_header":false, "users":[ { "name":"random-users", "uuid":"UUID", "alterId":0 }, { "name":"random-vws", "uuid":"$UUID", "alterId":0 } ], "transport":{ "type":"ws", "path":"$PATH_VMESS_WS" } } ] ```

Server and client log file

```console Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: panic: repeated read on failed websocket connection Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: goroutine 67516 [running]: Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/websocket.(*Conn).NextReader(0xc005d74420) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc0007343f0, {0xc000f460f8, 0x2, 0x2}) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325045037-5a86bb01e9f6/transport/v2raywebsocket/conn.go:45 +0x5e Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: io.ReadAtLeast({0x7f35c60fd5d8, 0xc0007343f0}, {0xc000f460f8, 0x2, 0x2}, 0x2) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/usr/local/go/src/io/io.go:332 +0x9a Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: io.ReadFull(...) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/usr/local/go/src/io/io.go:351 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: encoding/binary.Read({0x7f35c60fd5d8, 0xc0007343f0}, {0x1557a00, 0x1e5db48}, {0x111a860?, 0xc000f460f6}) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/usr/local/go/src/encoding/binary/binary.go:233 +0xc8 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing-vmess.(*StreamChunkReader).Read(0xc001408ae0, {0xc000f3e000, 0x4000, 0x4000}) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.3/chunk_length_stream.go:36 +0x89 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing/common/bufio.(*ExtendedReaderWrapper).ReadBuffer(0xc00cd5fa20?, 0xc001409770) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/conn.go:141 +0x5d Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing-vmess.(*AEADReader).ReadBuffer(0xc00c87bcc0, 0xc001409770) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.3/chunk_aead.go:54 +0x35 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing-vmess.(*serverPacketConn).ReadPacket(0xc00021b720, 0xb1da10a1f9?) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.3/service.go:521 +0x42 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xa?, 0x1e2c580?) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/cache.go:186 +0x13e Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing-box/experimental/trackerconn.(*HookPacketConn).ReadPacket(0xc00081c160, 0xc001409770) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325045037-5a86bb01e9f6/experimental/trackerconn/packet_conn.go:58 +0x58 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc00c87be80, 0xc00cd5fe80?) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/canceler/packet_timeout.go:35 +0xa8 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f35c604a240, 0xc011177740}, {0x7f35c59cc060?, 0xc00c87be80?}) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:260 +0x4ab Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing/common/bufio.CopyPacketConn.func1({0x1e5d700?, 0xc00cd5ff68?}) Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:340 +0x6c Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: github.com/sagernet/sing/common/task.(*Group).Run.func1() Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:59 +0x93 Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: created by github.com/sagernet/sing/common/task.(*Group).Run Mar 25 17:17:47 rucloud-dxdrn sing-box[24178]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:58 +0x225 ```
nekohasekai commented 1 year ago

Please provide complete configuration files and logs, and how to reproduce this issue.

malikshi commented 1 year ago

Please provide complete configuration files and logs, and how to reproduce this issue.

i updated the issues could you check it.

nekohasekai commented 1 year ago

Can you identify which version (or commit) introduced this problem?

malikshi commented 1 year ago

Can you identify which version (or commit) introduced this problem?

sorry i don't know the commit where is started, my current build is using cc34b6f it's last runing yesterday 26/03/2023 18.00 GMT +7 . i am using simple bash to check latest commit hourly and build it if it has new commit go install -v -tags "with_clash_api with_grpc with_wireguard with_utls with_gvisor with_quic with_dhcp with_ech with_reality_server" "github.com/sagernet/sing-box/cmd/sing-box@$COMMIT_ID"

malikshi commented 1 year ago

i saw kind of different logs for each protocol, vmess/vless/trojan i had running 3 of them with transport protocol ws Vless

Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: panic: repeated read on failed websocket connection
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: goroutine 47133 [running]:
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/websocket.(*Conn).NextReader(0xc00963b1e0)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc00760d410, {0xc0002a0108, 0x2, 0x2})
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/transport/v2raywebsocket/conn.go:45 +0x5e
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: io.ReadAtLeast({0x7f14288785b0, 0xc00760d9e0}, {0xc0002a0108, 0x2, 0x2}, 0x2)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/usr/local/go/src/io/io.go:332 +0x9a
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: io.ReadFull(...)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/usr/local/go/src/io/io.go:351
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: encoding/binary.Read({0x7f14288785b0, 0xc00760d9e0}, {0x1557e70, 0x1e5eb48}, {0x111a880?, 0xc0002a0106})
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/usr/local/go/src/encoding/binary/binary.go:233 +0xc8
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing-box/transport/vless.(*serverPacketConn).ReadPacket(0xc00b86e540, 0x0?)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/transport/vless/service.go:177 +0x8e
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xc00bc27ea0?, 0x1e2d580?)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/cache.go:186 +0x13e
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing-box/experimental/trackerconn.(*HookPacketConn).ReadPacket(0xc007c26e80, 0xc000350de0)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/experimental/trackerconn/packet_conn.go:58 +0x58
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc00293d9c0, 0xc007f1fe80?)
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/canceler/packet_timeout.go:35 +0xa8
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f14288d36f0, 0xc00ad17550}, {0x7f142841bf58?, 0xc00293d9c0?})
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:260 +0x4ab
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing/common/bufio.CopyPacketConn.func1({0x1e5e700?, 0xc007f1ff68?})
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:340 +0x6c
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: github.com/sagernet/sing/common/task.(*Group).Run.func1()
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:59 +0x93
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: created by github.com/sagernet/sing/common/task.(*Group).Run
Mar 26 09:07:07 rucloud-dxdrn sing-box[46829]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:58 +0x225

vmess

Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: panic: repeated read on failed websocket connection
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: goroutine 8750 [running]:
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/websocket.(*Conn).NextReader(0xc00a5de9a0)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc00891d740, {0xc00cd20ab8, 0x2, 0x2})
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/transport/v2raywebsocket/conn.go:45 +0x5e
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: io.ReadAtLeast({0x7fc9990d68f8, 0xc00891d740}, {0xc00cd20ab8, 0x2, 0x2}, 0x2)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/usr/local/go/src/io/io.go:332 +0x9a
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: io.ReadFull(...)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/usr/local/go/src/io/io.go:351
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: encoding/binary.Read({0x7fc9990d68f8, 0xc00891d740}, {0x1557e70, 0x1e5eb48}, {0x111a880?, 0xc00cd20ab6})
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/usr/local/go/src/encoding/binary/binary.go:233 +0xc8
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing-vmess.(*StreamChunkReader).Read(0xc00891dec0, {0xc000850000, 0x4000, 0x4000})
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.3/chunk_length_stream.go:36 +0x89
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing/common/bufio.(*ExtendedReaderWrapper).ReadBuffer(0xc002fd1a20?, 0xc007470690)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/conn.go:141 +0x5d
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing-vmess.(*AEADReader).ReadBuffer(0xc00367a100, 0xc007470690)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.3/chunk_aead.go:54 +0x35
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing-vmess.(*serverPacketConn).ReadPacket(0xc0004145a0, 0x189ca434c1b?)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.3/service.go:521 +0x42
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xc002fd1ba0?, 0x1e2d580?)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/cache.go:186 +0x13e
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing-box/experimental/trackerconn.(*HookPacketConn).ReadPacket(0xc008ee0700, 0xc007470690)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/experimental/trackerconn/packet_conn.go:58 +0x58
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc00367a180, 0xc002fd1e80?)
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/canceler/packet_timeout.go:35 +0xa8
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7fc9990ef108, 0xc0000130c0}, {0x7fc998c95f80?, 0xc00367a180?})
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:260 +0x4ab
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing/common/bufio.CopyPacketConn.func1({0xd81371?, 0xd7ed59?})
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:340 +0x6c
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: github.com/sagernet/sing/common/task.(*Group).Run.func1()
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:59 +0x93
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: created by github.com/sagernet/sing/common/task.(*Group).Run
Mar 26 09:35:05 rucloud-dxdrn sing-box[47561]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:58 +0x225

trojan

Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: panic: repeated read on failed websocket connection
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: goroutine 222324 [running]:
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/websocket.(*Conn).NextReader(0xc00383db80)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc009efd620, {0xc0051778e4, 0x1, 0x1})
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/transport/v2raywebsocket/conn.go:45 +0x5e
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: io.ReadAtLeast({0x7fd56deb5a28, 0xc009efd620}, {0xc0051778e4, 0x1, 0x1}, 0x1)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/usr/local/go/src/io/io.go:332 +0x9a
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: io.ReadFull(...)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/usr/local/go/src/io/io.go:351
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/rw.ReadByte({0x7fd56deb5a28, 0xc009efd620})
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/rw/read.go:22 +0x67
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/metadata.(*Serializer).ReadAddress(0xc00011c0c0, {0x7fd56deb5a28, 0xc009efd620})
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/metadata/serializer.go:120 +0x67
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/metadata.(*Serializer).ReadAddrPort(0xc00011c0c0, {0x7fd56deb5a28, 0xc009efd620})
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/metadata/serializer.go:167 +0x74
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing-box/transport/trojan.ReadPacket({0x155ac70?, 0xc009efd620}, 0x120c820?)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/transport/trojan/protocol.go:291 +0x8a
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing-box/transport/trojan.(*PacketConn).ReadPacket(0xc1000fdac5171fac?, 0x3427c1f519a?)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/transport/trojan/service.go:129 +0x3f
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xc00cec9ba0?, 0x1e2d580?)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/cache.go:186 +0x13e
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing-box/experimental/trackerconn.(*HookPacketConn).ReadPacket(0xc00032ae60, 0xc00952a7e0)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.1-0.20230325102036-cc34b6fa99dd/experimental/trackerconn/packet_conn.go:58 +0x58
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc00866e0c0, 0xc00cec9e80?)
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/canceler/packet_timeout.go:35 +0xa8
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7fd56dc792e0, 0xc00a1f4a20}, {0x7fd56d8a3700?, 0xc00866e0c0?})
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:260 +0x4ab
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/bufio.CopyPacketConn.func1({0x1e2d580?, 0xd7ed59?})
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:340 +0x6c
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: github.com/sagernet/sing/common/task.(*Group).Run.func1()
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:59 +0x93
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: created by github.com/sagernet/sing/common/task.(*Group).Run
Mar 26 11:19:40 rucloud-dxdrn sing-box[48882]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:58 +0x225
malikshi commented 1 year ago

i just updated the other 2 server to 0e37449 and got me same panic also i don't think isn't fault of my reverse nginx config.

edited: i will try the commit one by one to check where is the commit without panic, follow up later

malikshi commented 1 year ago

I tested every commit from latest to old commits to check where is the problem started at. I found this commit https://github.com/SagerNet/sing-box/commit/5dce72287929b8e43beb1b53bc3ab2f1e7392be7 work normally but since this commit https://github.com/SagerNet/sing-box/commit/4328c535a9e6d765e37a907787b50dc11a3b99ce problems started shown up. Could you check it sir @nekohasekai

nekohasekai commented 1 year ago

Can you give an example of a configuration run locally to reproduce the problem?

malikshi commented 1 year ago

i am sharing the account in my private groups so i can't tell what client app they're use, but mostly are using vmess/vless/trojan websocket 443 either SNI(allow insecure true)/over cdn cloudflare with clash/clash.meta on openwrt run in openclash. the error happens in server side but the i didn't see the panic log /var/log/box.log but i see the panic log in /var/log/syslog. i use sing-box client normally without problem in raspberry pi with today's commit 5647b32fbd8e7aa26571c1d069be034856a41426 . i am sorry if the information aren't enough, my server side config are same as initial issues opened and the panic error only at serverside. do you still needed the config running in my local raspberry pi? Edited: local raspberry pi config

{
  "log": {
    "disabled": false,
    "level": "debug",
    "output": "/var/log/box.log",
    "timestamp": true
  },
  "ntp": {
    "enabled": true,
    "server": "time.google.com",
    "server_port": 123,
    "interval": "30m",
    "detour": "WiFi"
  },
  "dns": {
    "servers": [
      {
        "tag": "adguardhome",
        "address": "192.168.100.1",
        "detour": "UMUM"
      },
      {
        "tag": "time",
        "address": "1.0.0.1",
        "strategy": "ipv4_only",
        "detour": "WiFi"
      },
      {
        "tag": "dns-block",
        "address": "rcode://success"
      }
    ],
    "rules": [
      {
        "domain": [
          "time.google.com"
        ],
        "server": "WiFi"
      },
      {
        "domain_suffix": [
          ".arpa.",
          ".arpa"
        ],
        "server": "dns-block"
      }
    ],
    "final": "adguardhome",
    "strategy": "ipv4_only",
    "disable_cache": true,
    "disable_expire": true
  },
  "inbounds": [
    {
      "type": "tun",
      "tag": "tun-in",
      "interface_name": "vpn0",
      "inet4_address": "172.19.0.1/30",
      "mtu": 30000,
      "auto_route": true,
      "strict_route": true,
      "stack": "system",
      "sniff": true
    }
  ],
  "outbounds": [
    {
      "type": "selector",
      "tag": "UMUM",
      "outbounds": [
        "ISP-DMD",
        "WiFi"
      ]
    },
    {
      "type": "urltest",
      "tag": "ISP-DMD",
      "outbounds": [
        "DMD-ws",
        "DMD-vlws"
      ],
      "url": "https://detectportal.firefox.com/success.txt",
      "interval": "86400s",
      "tolerance": 0
    },
    {
      "type": "selector",
      "tag": "BYPASS",
      "outbounds": [
        "usb",
        "direct",
        "WiFi"
      ]
    },
    {
      "type": "selector",
      "tag": "REMOTE",
      "outbounds": [
        "BYPASS",
        "ISP-DMD"
      ]
    },
    {
      "type": "selector",
      "tag": "GAMES",
      "outbounds": [
        "UMUM",
        "BYPASS"
      ]
    },
    {
      "type": "selector",
      "tag": "Parent",
      "outbounds": [
        "UMUM",
        "block"
      ]
    },
    {
      "type": "selector",
      "tag": "Mother",
      "outbounds": [
        "UMUM",
        "block"
      ]
    },
    {
      "type": "selector",
      "tag": "Children",
      "outbounds": [
        "UMUM",
        "block"
      ]
    },
    {
      "type": "vless",
      "tag": "DMD-vlws",
      "server": "domain",
      "server_port": 443,
      "uuid": "uuid",
      "flow": "",
      "tcp_fast_open": true,
      "connect_timeout": "3s",
      "tls": {
        "enabled": true,
        "disable_sni": false,
        "server_name": "domain",
        "insecure": true,
        "utls": {
          "enabled": true,
          "fingerprint": "android"
        }
      },
      "packet_encoding": "xudp",
      "transport": {
        "type": "ws",
        "path": "/vless-ws",
        "headers": {
          "Host": "domain"
        }
      },
      "detour": "direct"
    },
    {
      "type": "vmess",
      "tag": "DMD-ws",
      "server": "domain",
      "server_port": 443,
      "uuid": "uuid",
      "security": "none",
      "alter_id": 0,
      "global_padding": false,
      "authenticated_length": true,
      "tcp_fast_open": true,
      "connect_timeout": "3s",
      "tls": {
        "enabled": true,
        "disable_sni": false,
        "server_name": "domain",
        "insecure": true,
        "utls": {
          "enabled": true,
          "fingerprint": "android"
        }
      },
      "packet_encoding": "",
      "transport": {
        "type": "ws",
        "path": "/vmess-ws",
        "headers": {
          "Host": "domain"
        }
      },
      "multiplex": {
        "enabled": true,
        "protocol": "yamux",
        "max_streams": 32
      },
      "detour": "direct"
    },
    {
      "type": "direct",
      "tag": "direct",
      "bind_interface": "wan"
    },
    {
      "type": "direct",
      "tag": "usb",
      "bind_interface": "eth0"
    },
    {
      "type": "direct",
      "tag": "WiFi",
      "bind_interface": "wlan0"
    },
    {
      "type": "block",
      "tag": "block"
    },
    {
      "type": "dns",
      "tag": "dns-out"
    }
  ],
  "route": {
    "auto_detect_interface": false,
    "geoip": {
      "path": "/etc/sing-box/geoip.db",
      "download_url": "https://github.com/malikshi/sing-box-geo/releases/latest/download/geoip.db",
      "download_detour": "UMUM"
    },
    "geosite": {
      "path": "/etc/sing-box/geosite.db",
      "download_url": "https://github.com/malikshi/sing-box-geo/releases/latest/download/geosite.db",
      "download_detour": "UMUM"
    },
    "rules": [
      {
        "port": [
          22,
          9090
        ],
        "outbound": "REMOTE"
      },
      {
        "geosite": [
          "openai"
        ],
        "outbound": "REMOTE"
      },
      {
        "source_ip_cidr": [
          "192.168.100.5"
        ],
        "outbound": "Parent"
      },
      {
        "source_ip_cidr": [
          "192.168.100.9"
        ],
        "outbound": "Mother"
      },
      {
        "source_ip_cidr": [
          "192.168.100.10"
        ],
        "outbound": "Children"
      },
      {
        "network": "udp",
        "port_range": [
          "1023:65535"
        ],
        "outbound": "GAMES"
      },
      {
        "outbound": "dns-out",
        "port": [
          53
        ]
      },
      {
        "ip_cidr": [
          "224.0.0.0/3",
          "ff00::/8"
        ],
        "outbound": "block",
        "source_ip_cidr": [
          "224.0.0.0/3",
          "ff00::/8"
        ]
      }
    ],
    "final": "UMUM"
  },
  "experimental": {
    "clash_api": {
      "external_controller": "0.0.0.0:9090",
      "external_ui": "/etc/sing-box/yacd-gh-pages/",
      "secret": "abcdsef",
      "default_mode": "rule",
      "store_selected": true,
      "cache_file": "/etc/sing-box/cache.db"
    }
  }
}
nekohasekai commented 1 year ago

So you can't create a standalone config to reproduce the bug?

malikshi commented 1 year ago

just close the issues, i can't provided file log of client since mostly users are using clash/clash.meta, and my local sing-box client on raspberry pi won't give me panic errors. sorry for trouble, maybe in the futures someone have same problems and provide more information about it. i will keep using https://github.com/SagerNet/sing-box/commit/5dce72287929b8e43beb1b53bc3ab2f1e7392be7 for time being nor changes to other ray server

nekohasekai commented 1 year ago

Configuration files for clash (/meta) are also accepted if they can reproduce the problem.

kushim110 commented 1 year ago

@nekohasekai

I have the same problem.

it only happens when using different cores for server and client on this setup.

VLESS + TLS + Websocket

for me this is the setup when this error occurs.

Server Core (sing-box) <-> caddy (tls handshake) <-> Client Core (Xray[v2rayNG])

As mentioned in the docs I've tried v2rayNG as xray client and made sure to set early_data_header_name correctly to Sec-WebSocket-Protocol to make sing-box and Xray VLESS compatible and indeed it works and connects BUT only for a short period of time before sing-box panics and it's service restarts. the error message is something like this:

Mar 28 14:18:09 sing sing-box[31434]: +0000 2023-03-28 14:18:09 INFO [1631922953] outbound/direct[direct]: outbound connection to mtalk.google.com:5228
Mar 28 14:18:12 sing sing-box[31434]: panic: repeated read on failed websocket connection
Mar 28 14:18:12 sing sing-box[31434]: goroutine 52 [running]:
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/websocket.(*Conn).NextReader(0xc0001582c0)
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc0001dc810, {0xc00009d3f8, 0x2, 0x2})
Mar 28 14:18:12 sing sing-box[31434]:         /Users/sekai/Projects/sing-box/transport/v2raywebsocket/conn.go:45 +0x5e
Mar 28 14:18:12 sing sing-box[31434]: io.ReadAtLeast({0x7f6687cae9d0, 0xc00027c000}, {0xc00009d3f8, 0x2, 0x2}, 0x2)
Mar 28 14:18:12 sing sing-box[31434]:         $GOROOT/src/io/io.go:332 +0x9a
Mar 28 14:18:12 sing sing-box[31434]: io.ReadFull(...)
Mar 28 14:18:12 sing sing-box[31434]:         $GOROOT/src/io/io.go:351
Mar 28 14:18:12 sing sing-box[31434]: encoding/binary.Read({0x7f6687cae9d0, 0xc00027c000}, {0x123ef00, 0x1996b70}, {0xebd4c0?, 0xc00009d3f6})
Mar 28 14:18:12 sing sing-box[31434]:         $GOROOT/src/encoding/binary/binary.go:233 +0xc8
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing-box/transport/vless.(*serverPacketConn).ReadPacket(0xc00007d020, 0xc000091c00?)
Mar 28 14:18:12 sing sing-box[31434]:         /Users/sekai/Projects/sing-box/transport/vless/service.go:177 +0x8e
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xc100dbabae433d42?, 0x39a69d0dfce?)
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/cache.go:186 +0x13e
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc00003b180, 0xc000091e80?)
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/sing@v0.2.1/common/canceler/packet_timeout.go:35 +0xa8
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f6687cf2c58, 0xc00000e2f8}, {0x7f6687cf2c78?, 0xc00003b180?})
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:260 +0x4ab
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing/common/bufio.CopyPacketConn.func1({0xc000122701?, 0xc0001227d0?})
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/sing@v0.2.1/common/bufio/copy.go:340 +0x6c
Mar 28 14:18:12 sing sing-box[31434]: github.com/sagernet/sing/common/task.(*Group).Run.func1()
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:59 +0x93
Mar 28 14:18:12 sing sing-box[31434]: created by github.com/sagernet/sing/common/task.(*Group).Run
Mar 28 14:18:12 sing sing-box[31434]:         pkg/mod/github.com/sagernet/sing@v0.2.1/common/task/task.go:58 +0x225
Mar 28 14:18:12 sing systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 28 14:18:12 sing systemd[1]: sing-box.service: Failed with result 'exit-code'.

But it does NOT happen when using sing-box client with the same configuration.

sing-box version: 1.2.1

sing-box server config

{
  "log": {
    "level": "info",
    "timestamp": true
  },
  "route": {
    "geoip": {
      "path": "geoip.db",
      "download_url": "https://github.com/soffchen/sing-geoip/releases/latest/download/geoip.db"
    },
    "geosite": {
      "path": "geosite.db",
      "download_url": "https://github.com/soffchen/sing-geosite/releases/latest/download/geosite.db"
    },
    "rules": [
      {
        "geoip": ["cn", "private"],
        "outbound": "block"
      }
    ]
  },
  "inbounds": [
    {
      "type": "vless",
      "tag": "vless-in",
      "listen": "127.0.0.1",
      "listen_port": 10002,
      "sniff": true,
      "sniff_override_destination": true,
      "domain_strategy": "ipv4_only",
      "users": [
        {
          "name": "sekai",
          "uuid": "bf000d23-0752-40b4-affe-68f7707a9661",
          "flow": ""
        }
      ],
      "transport": {
        "type": "ws",
        "path": "/ws-path",
        "early_data_header_name": "Sec-WebSocket-Protocol"
      }
    }
  ],
  "outbounds": [
    {
      "type": "direct",
      "tag": "direct"
    },
    {
      "type": "block",
      "tag": "block"
    }
  ]
}

sing-box client config:

{
  "log": {
    "level": "info",
    "timestamp": true
  },
  "inbounds": [
    {
      "type": "mixed",
      "listen": "::",
      "listen_port": 10000,
      "domain_strategy": "ipv4_only"
    }
  ],
  "outbounds": [
    {
      "type": "vless",
      "tag": "proxy",
      "server": "subdomain.domain.tld",
      "server_port": 443,
      "uuid": "bf000d23-0752-40b4-affe-68f7707a9661",
      "flow": "",
      "network": "tcp",
      "tls": {
        "enabled": true,
        "server_name": "subdomain.domain.tld"
      },
      "packet_encoding": "xudp",
      "transport": {
        "type": "ws",
        "path": "/ws-path",
        "early_data_header_name": "Sec-WebSocket-Protocol"
      }
    },
    {
      "type": "direct",
      "tag": "direct"
    },
    {
      "type": "block",
      "tag": "block"
    }
  ]
}

v2rayNG URI config:

vless://bf000d23-0752-40b4-affe-68f7707a9661@subdomain.domain.tld:443?path=%2Fws-path&security=tls&encryption=none&alpn=http/1.1&host=subdomain.domain.tld&fp=chrome&type=ws&sni=subdomain.domain.tld#Websocket-Test

Caddyfile Config:

{
    order reverse_proxy before route
    email hello@there.loc
}

subdomain.domain.tld {
    @vlws {
        path /ws-path
        header Connection *Upgrade*
        header Upgrade websocket
    }
    reverse_proxy @vlws 127.0.0.1:10002

    @host {
        host subdomain.domain.tld
    }
    route @host {
        file_server {
            root /usr/share/caddy
        }
    }
}

I hope it helps you fix this problem. Many users can only access Xray clients and there are lots of good options in sing-box on server-side which helps.

malikshi commented 1 year ago

i will give a shot later after deploy new vm. and tried different client apps

nekohasekai commented 1 year ago

Try https://github.com/SagerNet/sing-box/commit/f8aa0dc7a36bbc26cc96c10ac8a38fe81e81fb78

malikshi commented 1 year ago

i will give a shot later after deploy new vm. and tried different client apps

still don't have new vm.

Try f8aa0dc

updated in VMs production, and has been running for 2 hours without panic, i will do follow up info at peak hours.

Edited

Active: active (running) since Sat 2023-04-08 10:36:47 WIB; 10h ago no issues so far in 3 VMs. i don't get error/panic also no reports disconnected because server restart from my groups, i think the commits solved the problems.

malikshi commented 1 year ago

@nekohasekai sir i want to ask if the fix code already merged in latest commit? i got an error after updated to this commit aceb82a

Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: panic: repeated read on failed websocket connection
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: goroutine 31469 [running]:
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/websocket.(*Conn).NextReader(0xc00388a580)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc00a86f080, {0xc00b6be408, 0x1, 0x1})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.4-0.20230409145306-aceb82a75e96/transport/v2raywebsocket/conn.go:45 +0x5e
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio/deadline.(*Reader).Read(0xc0093e2510, {0xc00b6be408, 0x1, 0x1})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/deadline/reader.go:40 +0xdb
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio/deadline.(*Conn).Read(0xc00a489540?, {0xc00b6be408?, 0xc00a489528?, 0x40fa07?})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/deadline/conn.go:22 +0x25
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: io.ReadAtLeast({0x7f030b7e38b0, 0xc0081ca420}, {0xc00b6be408, 0x1, 0x1}, 0x1)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/usr/local/go/src/io/io.go:332 +0x9a
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: io.ReadFull(...)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/usr/local/go/src/io/io.go:351
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/rw.ReadByte({0x7f030b7e38b0, 0xc0081ca420})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/rw/read.go:22 +0x67
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/metadata.(*Serializer).ReadAddress(0xc0001220c0, {0x7f030b7e38b0, 0xc0081ca420})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/metadata/serializer.go:120 +0x67
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/metadata.(*Serializer).ReadAddrPort(0xc0001220c0, {0x7f030b7e38b0, 0xc0081ca420})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/metadata/serializer.go:167 +0x74
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing-box/transport/trojan.ReadPacket({0x1585af0?, 0xc0081ca420}, 0x122dc20?)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.4-0.20230409145306-aceb82a75e96/transport/trojan/protocol.go:291 +0x8a
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing-box/transport/trojan.(*PacketConn).ReadPacket(0x10?, 0x11e9700?)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.4-0.20230409145306-aceb82a75e96/transport/trojan/service.go:130 +0x3f
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).ReadPacket(0xc000281b80, 0xc00a88b830)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/deadline/packet_reader.go:89 +0xc2
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio/deadline.(*PacketConn).ReadPacket(0x0?, 0xc00a489c00?)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/deadline/packet_conn.go:26 +0x33
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xc1050c9fde36e138?, 0x53ee2fa5bb?)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/cache.go:186 +0x13e
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc005f98a40, 0xc00a489e80?)
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/canceler/packet_timeout.go:35 +0xa8
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f030b719868, 0xc0067d6a38}, {0x7f030b2bffe0?, 0xc005f98a40?})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/copy.go:260 +0x4ab
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/bufio.CopyPacketConn.func1({0x1e9ffc0?, 0xc00a489f68?})
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/bufio/copy.go:340 +0x6c
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: github.com/sagernet/sing/common/task.(*Group).Run.func1()
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/task/task.go:59 +0x93
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: created by github.com/sagernet/sing/common/task.(*Group).Run
Apr 10 14:29:52 rucloud-dxdrn sing-box[35352]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230409145127-a82d82e55914/common/task/task.go:58 +0x225
nekohasekai commented 1 year ago

Try https://github.com/SagerNet/sing-box/commit/e35c325e75453f38f7307596b9e88b797af10c00

malikshi commented 1 year ago

Try e35c325

updated and got the same error https://github.com/SagerNet/sing-box/issues/473#issuecomment-1483992124 but i see only error from trojan i see no panic error from vmess/vless

Apr 11 11:15:40 cpanel1 sing-box[26061]: panic: repeated read on failed websocket connection
Apr 11 11:15:40 cpanel1 sing-box[26061]: goroutine 198622 [running]:
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/websocket.(*Conn).NextReader(0xc00dd57e40)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc00063fc80, {0xc0069cd7f8, 0x1, 0x1})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.4-0.20230411023544-e35c325e7545/transport/v2raywebsocket/conn.go:45 +0x5e
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio/deadline.(*Reader).Read(0xc010700870, {0xc0069cd7f8, 0x1, 0x1})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/deadline/reader.go:40 +0xdb
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio/deadline.(*Conn).Read(0xc00d8ef568?, {0xc0069cd7f8?, 0xc00d8ef550?, 0x40fa07?})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/deadline/conn.go:22 +0x25
Apr 11 11:15:40 cpanel1 sing-box[26061]: io.ReadAtLeast({0x7f74d855e200, 0xc0098aa480}, {0xc0069cd7f8, 0x1, 0x1}, 0x1)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/usr/local/go/src/io/io.go:332 +0x9a
Apr 11 11:15:40 cpanel1 sing-box[26061]: io.ReadFull(...)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/usr/local/go/src/io/io.go:351
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/rw.ReadByte({0x7f74d855e200, 0xc0098aa480})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/rw/read.go:22 +0x67
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/metadata.(*Serializer).ReadAddress(0xc0001220c0, {0x7f74d855e200, 0xc0098aa480})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/metadata/serializer.go:120 +0x67
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/metadata.(*Serializer).ReadAddrPort(0xc0001220c0, {0x7f74d855e200, 0xc0098aa480})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/metadata/serializer.go:167 +0x74
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing-box/transport/trojan.ReadPacket({0x1588c10?, 0xc0098aa480}, 0xb34?)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.4-0.20230411023544-e35c325e7545/transport/trojan/protocol.go:291 +0x8a
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing-box/transport/trojan.(*PacketConn).ReadPacket(0x10?, 0x11ebc20?)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.4-0.20230411023544-e35c325e7545/transport/trojan/service.go:130 +0x3f
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).ReadPacket(0xc012abdef0, 0xc007c59e00)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/deadline/packet_reader.go:89 +0xc2
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio/deadline.(*PacketConn).ReadPacket(0x0?, 0xc00d8efc28?)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/deadline/packet_conn.go:26 +0x33
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio.(*CachedPacketConn).ReadPacket(0xc105559ec5e40b4c?, 0x16db606d0e5?)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/cache.go:186 +0x13e
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/canceler.(*TimeoutPacketConn).ReadPacket(0xc010cffc80, 0xc00d8efea8?)
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/canceler/packet_timeout.go:36 +0xa8
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio.CopyPacket({0x7f74d84ac4c8, 0xc003a483f0}, {0x7f74d84c04c8?, 0xc010cffc80?})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/copy.go:264 +0x4ab
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/bufio.CopyPacketConnContextList.func1({0x472ad7?, 0xc010950050?})
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/bufio/copy.go:348 +0x6c
Apr 11 11:15:40 cpanel1 sing-box[26061]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/task/task.go:68 +0x84
Apr 11 11:15:40 cpanel1 sing-box[26061]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList
Apr 11 11:15:40 cpanel1 sing-box[26061]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230411023500-28b06822071a/common/task/task.go:67 +0x17e
Apr 11 11:15:40 cpanel1 systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 11 11:15:40 cpanel1 systemd[1]: sing-box.service: Failed with result 'exit-code'.
Apr 11 11:15:40 cpanel1 systemd[1]: sing-box.service: Consumed 4min 36.752s CPU time.
Apr 11 11:15:41 cpanel1 systemd[1]: sing-box.service: Scheduled restart job, restart counter is at 3.
nekohasekai commented 1 year ago

Try https://github.com/SagerNet/sing-box/commit/f79828761e1fc45bda6bb6f195806efe04aa47bf

malikshi commented 1 year ago

Try f798287

solved the problems. No errors found

malikshi commented 1 year ago

i don't know if this related to this issues, but i got an panic error e640fcf207c96e985f7b91a84a059b99df1f0970

Apr 13 10:12:40 cpanel1 sing-box[11730]: panic: runtime error: slice bounds out of range [824695060352:0]
Apr 13 10:12:40 cpanel1 sing-box[11730]: goroutine 1779 [running]:
Apr 13 10:12:40 cpanel1 sing-box[11730]: github.com/sagernet/sing/common/buf.(*Buffer).FreeBytes(...)
Apr 13 10:12:40 cpanel1 sing-box[11730]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413023204-48b019b13e78/common/buf/buffer.go:378
Apr 13 10:12:40 cpanel1 sing-box[11730]: github.com/sagernet/sing/common/buf.(*Buffer).ReadOnceFrom(0x0, {0x1572900?, 0xc0003109f0?})
Apr 13 10:12:40 cpanel1 sing-box[11730]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413023204-48b019b13e78/common/buf/buffer.go:171 +0x9c
Apr 13 10:12:40 cpanel1 sing-box[11730]: github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).pipeReadPacket(0xc0002368f0, 0xc001303f00, 0x0?)
Apr 13 10:12:40 cpanel1 sing-box[11730]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413023204-48b019b13e78/common/bufio/deadline/packet_reader.go:107 +0x225
Apr 13 10:12:40 cpanel1 sing-box[11730]: github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).ReadPacket.func1()
Apr 13 10:12:40 cpanel1 sing-box[11730]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413023204-48b019b13e78/common/bufio/deadline/packet_reader.go:85 +0x77
Apr 13 10:12:40 cpanel1 sing-box[11730]: created by github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).ReadPacket
Apr 13 10:12:40 cpanel1 sing-box[11730]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413023204-48b019b13e78/common/bufio/deadline/packet_reader.go:84 +0x2ea
Apr 13 10:12:40 cpanel1 systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
nekohasekai commented 1 year ago

Try 8e1582a1ccac3265cd395d879d10392897806299

malikshi commented 1 year ago

Try 8e1582a

Apr 13 14:09:11 cpanel1 sing-box[20888]: panic: close of closed channel
Apr 13 14:09:11 cpanel1 sing-box[20888]: goroutine 268 [running]:
Apr 13 14:09:11 cpanel1 sing-box[20888]: github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).ReadPacket.func1()
Apr 13 14:09:11 cpanel1 sing-box[20888]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413062642-471c76893396/common/bufio/deadline/packet_reader.go:101 +0x128
Apr 13 14:09:11 cpanel1 sing-box[20888]: created by github.com/sagernet/sing/common/bufio/deadline.(*PacketReader).ReadPacket
Apr 13 14:09:11 cpanel1 sing-box[20888]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.3-0.20230413062642-471c76893396/common/bufio/deadline/packet_reader.go:99 +0x38e
nekohasekai commented 1 year ago

Try https://github.com/SagerNet/sing-box/commit/61e9a832ccec6e4f773d7b921ef921decac7eabe

malikshi commented 1 year ago

Try 61e9a83

try this 1f7430e and 9b4b4fc26e5beccd6bdd3271e9238e2a221bd6d8 both of them works

Edited i am also try 7f49c45 works fine too. i tried on 3 differents VM for each commit i mentioned

malikshi commented 1 year ago

since it's been while so i though recent commit already fix so i updated to 18f0564 there panic

Apr 18 15:27:53 cpanel1 sing-box[22317]: panic: repeated read on failed websocket connection
Apr 18 15:27:53 cpanel1 sing-box[22317]: goroutine 25267 [running]:
Apr 18 15:27:53 cpanel1 sing-box[22317]: github.com/sagernet/websocket.(*Conn).NextReader(0xc00870e160)
Apr 18 15:27:53 cpanel1 sing-box[22317]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Apr 18 15:27:53 cpanel1 sing-box[22317]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc0086038c0, {0xc004390000, 0x8000, 0x8000})
Apr 18 15:27:53 cpanel1 sing-box[22317]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.5-0.20230418072141-18f0564c5cb0/transport/v2raywebsocket/conn.go:45 +0x5e
Apr 18 15:27:53 cpanel1 sing-box[22317]: github.com/sagernet/sing/common/bufio.(*ExtendedReaderWrapper).ReadBuffer(0x1578f20?, 0xc002dcbb00)
Apr 18 15:27:53 cpanel1 sing-box[22317]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/conn.go:78 +0x5d
Apr 18 15:27:53 cpanel1 sing-box[22317]: github.com/sagernet/sing/common/bufio/deadline.(*reader).pipeReadBuffer(0xc007714840, 0x1169fe0?, 0x0)
Apr 18 15:27:53 cpanel1 sing-box[22317]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/deadline/reader.go:135 +0x54
Apr 18 15:27:53 cpanel1 sing-box[22317]: created by github.com/sagernet/sing/common/bufio/deadline.(*reader).ReadBuffer
Apr 18 15:27:53 cpanel1 sing-box[22317]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/deadline/reader.go:103 +0xc5
Apr 18 15:27:53 cpanel1 systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 18 15:27:53 cpanel1 systemd[1]: sing-box.service: Failed with result 'exit-code'.

Apr 18 15:28:04 cpanel1 sing-box[22844]: panic: repeated read on failed websocket connection
Apr 18 15:28:04 cpanel1 sing-box[22844]: goroutine 5905 [running]:
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/websocket.(*Conn).NextReader(0xc0008ec840)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/websocket@v0.0.0-20220913015213-615516348b4e/conn.go:1010 +0x2a8
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*WebsocketConn).Read(0xc000291200, {0xc004bf3e68, 0x2, 0x2})
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing-box@v1.2.5-0.20230418072141-18f0564c5cb0/transport/v2raywebsocket/conn.go:45 +0x5e
Apr 18 15:28:04 cpanel1 sing-box[22844]: io.ReadAtLeast({0x7f5f91849d78, 0xc000291200}, {0xc004bf3e68, 0x2, 0x2}, 0x2)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/usr/local/go/src/io/io.go:332 +0x9a
Apr 18 15:28:04 cpanel1 sing-box[22844]: io.ReadFull(...)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/usr/local/go/src/io/io.go:351
Apr 18 15:28:04 cpanel1 sing-box[22844]: encoding/binary.Read({0x7f5f91849d78, 0xc000291200}, {0x158fa10, 0x1eb3548}, {0x11437e0?, 0xc004bf3e66})
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/usr/local/go/src/encoding/binary/binary.go:233 +0xc8
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing-vmess.(*StreamChunkReader).Read(0xc000fb1230, {0xc003dde000, 0x8000, 0x8000})
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.5-0.20230417103030-8c3070ae3fb3/chunk_length_stream.go:36 +0x89
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing/common/bufio.(*ExtendedReaderWrapper).ReadBuffer(0xc004d5a7e0?, 0xc004d5a7e0)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/conn.go:78 +0x5d
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing-vmess.(*AEADReader).ReadBuffer(0xc0002ec8c0, 0xc004d5a7e0)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.5-0.20230417103030-8c3070ae3fb3/chunk_aead.go:54 +0x35
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing/common/bufio.(*ChunkReader).ReadBuffer(0xc003dde000?, 0x1?)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/chunk.go:26 +0x150
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing-vmess.(*serverConn).ReadBuffer(0xc002b13f58?, 0xc0034addc0?)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing-vmess@v0.1.5-0.20230417103030-8c3070ae3fb3/service.go:470 +0x22
Apr 18 15:28:04 cpanel1 sing-box[22844]: github.com/sagernet/sing/common/bufio/deadline.(*reader).pipeReadBuffer(0xc0011e4f00, 0xc000125308?, 0x0)
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/deadline/reader.go:135 +0x54
Apr 18 15:28:04 cpanel1 sing-box[22844]: created by github.com/sagernet/sing/common/bufio/deadline.(*reader).ReadBuffer
Apr 18 15:28:04 cpanel1 sing-box[22844]: #011/root/go/pkg/mod/github.com/sagernet/sing@v0.2.4-0.20230418025125-f196b4303e31/common/bufio/deadline/reader.go:103 +0xc5
Apr 18 15:28:04 cpanel1 systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Apr 18 15:28:04 cpanel1 systemd[1]: sing-box.service: Failed with result 'exit-code'.
nekohasekai commented 1 year ago

https://github.com/SagerNet/sing-box/commit/f98de04fdd5bee961f96e3e778bbc9acd7d8c949

malikshi commented 1 year ago

f98de04

Thanks you! it's fixed. there no panic in 3 VMs so far

malikshi commented 1 year ago

commit b0a978d sorry i got panic again.

May  2 17:05:18 runcloud-ikd sing-box[23944]: panic: buffer overflow: cap 32768,start 0, need 6
May  2 17:05:18 runcloud-ikd sing-box[23944]: goroutine 2381 [running]:
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/buf.(*Buffer).ExtendHeader(0xc0021ddd18, 0x6)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/buf/buffer.go:147 +0x1b9
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*Writer).WriteBuffer(0xc002f92c60, 0xc0021ddd18)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/v2raywebsocket/writer.go:54 +0xca
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.(*ExtendedConnWrapper).WriteBuffer(0x1016b00?, 0xc002b9f530?)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/conn.go:156 +0x22
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing-box/transport/trojan.(*ClientConn).WriteBuffer(0x13a5360?, 0xc001c78d00?)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/trojan/protocol.go:65 +0x175
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.CopyExtendedBuffer({0x7f99901a8970?, 0xc001c78d00}, {0x7f99901a89d8?, 0xc001c78d00}, {0x7f99901837a0, 0xc0009ff360}, 0xc0021ddd18, {0x0, 0x0, 0x0}, ...)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:113 +0x219
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.CopyExtended({0x7f99901a8970, 0xc001c78d00}, {0x7f99901a89d8?, 0xc001c78d00}, {0x7f99901837a0?, 0xc0009ff360}, {0x0, 0x0, 0x0}, {0x0, ...})
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:87 +0x4bf
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.Copy({0x7f99901a8970, 0xc001c78d00}, {0x7f9998037d08?, 0xc0009ff360?})
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:55 +0x5b3
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.CopyConnContextList.func1({0x43bba7?, 0xc0021ddf60?})
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:213 +0x75
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:68 +0x84
May  2 17:05:18 runcloud-ikd sing-box[23944]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:67 +0x17e
May  2 17:05:18 runcloud-ikd systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May  2 17:05:18 runcloud-ikd systemd[1]: sing-box.service: Failed with result 'exit-code'.
May  2 17:05:18 runcloud-ikd systemd[1]: sing-box.service: Consumed 2.453s CPU time.
May  2 17:05:19 runcloud-ikd systemd[1]: sing-box.service: Scheduled restart job, restart counter is at 18.

May  2 17:05:26 runcloud-ikd sing-box[23960]: panic: buffer overflow: cap 32768,start 0, need 6
May  2 17:05:26 runcloud-ikd sing-box[23960]: goroutine 1678 [running]:
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/buf.(*Buffer).ExtendHeader(0xc002b83d18, 0x6)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/buf/buffer.go:147 +0x1b9
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*Writer).WriteBuffer(0xc002569640, 0xc002b83d18)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/v2raywebsocket/writer.go:54 +0xca
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.(*ExtendedConnWrapper).WriteBuffer(0x1016b00?, 0xc001371dd0?)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/conn.go:156 +0x22
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing-box/transport/trojan.(*ClientConn).WriteBuffer(0x13a5360?, 0xc001f36a00?)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/trojan/protocol.go:65 +0x175
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.CopyExtendedBuffer({0x7f245820e440?, 0xc001f36a00}, {0x7f245820e4a8?, 0xc001f36a00}, {0x7f245819b028, 0xc000c2f900}, 0xc002b83d18, {0x0, 0x0, 0x0}, ...)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:113 +0x219
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.CopyExtended({0x7f245820e440, 0xc001f36a00}, {0x7f245820e4a8?, 0xc001f36a00}, {0x7f245819b028?, 0xc000c2f900}, {0x0, 0x0, 0x0}, {0x0, ...})
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:87 +0x4bf
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.Copy({0x7f245820e440, 0xc001f36a00}, {0x7f2458251908?, 0xc000c2f900?})
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:55 +0x5b3
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.CopyConnContextList.func1({0x16056c0?, 0x0?})
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:213 +0x75
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:68 +0x84
May  2 17:05:26 runcloud-ikd sing-box[23960]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:67 +0x17e
May  2 17:05:26 runcloud-ikd systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May  2 17:05:26 runcloud-ikd systemd[1]: sing-box.service: Failed with result 'exit-code'.
May  2 17:05:26 runcloud-ikd systemd[1]: sing-box.service: Consumed 2.181s CPU time.
May  2 17:05:27 runcloud-ikd systemd[1]: sing-box.service: Scheduled restart job, restart counter is at 19.
malikshi commented 1 year ago

commit b0a978d sorry i got panic again.

May  2 17:05:18 runcloud-ikd sing-box[23944]: panic: buffer overflow: cap 32768,start 0, need 6
May  2 17:05:18 runcloud-ikd sing-box[23944]: goroutine 2381 [running]:
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/buf.(*Buffer).ExtendHeader(0xc0021ddd18, 0x6)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/buf/buffer.go:147 +0x1b9
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*Writer).WriteBuffer(0xc002f92c60, 0xc0021ddd18)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/v2raywebsocket/writer.go:54 +0xca
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.(*ExtendedConnWrapper).WriteBuffer(0x1016b00?, 0xc002b9f530?)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/conn.go:156 +0x22
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing-box/transport/trojan.(*ClientConn).WriteBuffer(0x13a5360?, 0xc001c78d00?)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/trojan/protocol.go:65 +0x175
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.CopyExtendedBuffer({0x7f99901a8970?, 0xc001c78d00}, {0x7f99901a89d8?, 0xc001c78d00}, {0x7f99901837a0, 0xc0009ff360}, 0xc0021ddd18, {0x0, 0x0, 0x0}, ...)
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:113 +0x219
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.CopyExtended({0x7f99901a8970, 0xc001c78d00}, {0x7f99901a89d8?, 0xc001c78d00}, {0x7f99901837a0?, 0xc0009ff360}, {0x0, 0x0, 0x0}, {0x0, ...})
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:87 +0x4bf
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.Copy({0x7f99901a8970, 0xc001c78d00}, {0x7f9998037d08?, 0xc0009ff360?})
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:55 +0x5b3
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/bufio.CopyConnContextList.func1({0x43bba7?, 0xc0021ddf60?})
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:213 +0x75
May  2 17:05:18 runcloud-ikd sing-box[23944]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:68 +0x84
May  2 17:05:18 runcloud-ikd sing-box[23944]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList
May  2 17:05:18 runcloud-ikd sing-box[23944]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:67 +0x17e
May  2 17:05:18 runcloud-ikd systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May  2 17:05:18 runcloud-ikd systemd[1]: sing-box.service: Failed with result 'exit-code'.
May  2 17:05:18 runcloud-ikd systemd[1]: sing-box.service: Consumed 2.453s CPU time.
May  2 17:05:19 runcloud-ikd systemd[1]: sing-box.service: Scheduled restart job, restart counter is at 18.

May  2 17:05:26 runcloud-ikd sing-box[23960]: panic: buffer overflow: cap 32768,start 0, need 6
May  2 17:05:26 runcloud-ikd sing-box[23960]: goroutine 1678 [running]:
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/buf.(*Buffer).ExtendHeader(0xc002b83d18, 0x6)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/buf/buffer.go:147 +0x1b9
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing-box/transport/v2raywebsocket.(*Writer).WriteBuffer(0xc002569640, 0xc002b83d18)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/v2raywebsocket/writer.go:54 +0xca
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.(*ExtendedConnWrapper).WriteBuffer(0x1016b00?, 0xc001371dd0?)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/conn.go:156 +0x22
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing-box/transport/trojan.(*ClientConn).WriteBuffer(0x13a5360?, 0xc001f36a00?)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing-box@v1.2.7-0.20230501044530-b0a978d4b6dc/transport/trojan/protocol.go:65 +0x175
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.CopyExtendedBuffer({0x7f245820e440?, 0xc001f36a00}, {0x7f245820e4a8?, 0xc001f36a00}, {0x7f245819b028, 0xc000c2f900}, 0xc002b83d18, {0x0, 0x0, 0x0}, ...)
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:113 +0x219
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.CopyExtended({0x7f245820e440, 0xc001f36a00}, {0x7f245820e4a8?, 0xc001f36a00}, {0x7f245819b028?, 0xc000c2f900}, {0x0, 0x0, 0x0}, {0x0, ...})
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:87 +0x4bf
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.Copy({0x7f245820e440, 0xc001f36a00}, {0x7f2458251908?, 0xc000c2f900?})
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:55 +0x5b3
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/bufio.CopyConnContextList.func1({0x16056c0?, 0x0?})
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/bufio/copy.go:213 +0x75
May  2 17:05:26 runcloud-ikd sing-box[23960]: github.com/sagernet/sing/common/task.(*Group).RunContextList.func1()
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:68 +0x84
May  2 17:05:26 runcloud-ikd sing-box[23960]: created by github.com/sagernet/sing/common/task.(*Group).RunContextList
May  2 17:05:26 runcloud-ikd sing-box[23960]: #011github.com/sagernet/sing@v0.2.5-0.20230501044132-8365dd48a17a/common/task/task.go:67 +0x17e
May  2 17:05:26 runcloud-ikd systemd[1]: sing-box.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
May  2 17:05:26 runcloud-ikd systemd[1]: sing-box.service: Failed with result 'exit-code'.
May  2 17:05:26 runcloud-ikd systemd[1]: sing-box.service: Consumed 2.181s CPU time.
May  2 17:05:27 runcloud-ikd systemd[1]: sing-box.service: Scheduled restart job, restart counter is at 19.

sorry this one is my bad at list order of route rule setting