search

openimsdk / open-im-server

IM Chat
https://openim.io
Apache License 2.0
13.79k stars 2.43k forks source link

[BUG] 突发性报 invalid memory address or nil pointer dereference #2543

Open cipchk opened 3 weeks ago

cipchk commented 3 weeks ago

OpenIM Server Version

3.8.0

Operating System and CPU Architecture

Linux (AMD)

Deployment Method

Docker Deployment

Bug Description and Steps to Reproduce

无任何预警，直接 openim-server 报出以下错误，并该服务直接当机：

    2024-08-21 09:18:58.192 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]                          readMessage                                         {"remoteAddr": "172.30.0.1:36562", "platform": "Android", "connID": "661ccf00b8caaa109a37f39bf090e678", "operationID": "1724231852128687888", "opUserID": "10326", "messageType": "-1", "error": "read tcp 172.30.0.14:10001->172.30.0.1:36562: i/o timeout"}
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x11def43]
    goroutine 522741 [running]:
    github.com/openimsdk/open-im-server/v3/pkg/common/storage/cache/redis.(*msgCache).GetMessagesBySeq.func1({0x1923da0, 0xc0017fdd10}, 0xc000ddb2e7?, {0xc000af3320?, 0x32, 0x32})
        /openim-server/pkg/common/storage/cache/redis/msg.go:167 +0x1e3
    github.com/openimsdk/open-im-server/v3/pkg/common/storage/cache/redis.ProcessKeysBySlot.func1()
        /openim-server/pkg/common/storage/cache/redis/redis_shard_manager.go:182 +0x6d
    golang.org/x/sync/errgroup.(*Group).Go.func1()
        /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:78 +0x56
    created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 522739
        /go/pkg/mod/golang.org/x/sync@v0.6.0/errgroup/errgroup.go:75 +0x96
    2024-08-21 09:19:09.209 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - PullMessageBySeqs       {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724231943651", "opUserID": "10366", "funcName": "/openim.msg.msg/PullMessageBySeqs", "error": "rpc error: code = Unavailable desc = error reading from server: EOF"}
    2024-08-21 09:19:23.420 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - PullMessageBySeqs       {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724231963195", "opUserID": "10366", "funcName": "/openim.msg.msg/PullMessageBySeqs", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\""}
    2024-08-21 09:19:51.092 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724231993454859285", "opUserID": "10366", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:19:51.109 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724231990506", "opUserID": "10366", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:19:51.887 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724231993727428437", "opUserID": "10366", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:19:51.898 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724231990711", "opUserID": "10366", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:19:55.093 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724231997926597916", "opUserID": "10366", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:19:55.104 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724231994189", "opUserID": "10366", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:21:07.190 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - SearchMessage           {"platform": "Admin", "operationID": "063397fa-1861-4117-b8b3-4250f6da776e", "opUserID": "imAdmin", "funcName": "/openim.msg.msg/SearchMessage", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:21:11.763 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - SearchMessage           {"platform": "Admin", "operationID": "685dbe3b-1ee5-42c7-a43f-e839d54b57a0", "opUserID": "imAdmin", "funcName": "/openim.msg.msg/SearchMessage", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:21:30.719 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]                          readMessage                                         {"remoteAddr": "172.30.0.1:53304", "platform": "Android", "connID": "95df5691ae2661fc383c21a8c8fc1926", "operationID": "1724231550755696624", "opUserID": "10274", "messageType": "-1", "error": "read tcp 172.30.0.14:10001->172.30.0.1:53304: i/o timeout"}
    2024-08-21 09:21:53.188 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]                          readMessage                                         {"remoteAddr": "172.30.0.1:41364", "platform": "Android", "connID": "453909c5e790092fc8eeb1d23033f71a", "operationID": "1724231504312537961", "opUserID": "10300", "messageType": "-1", "error": "read tcp 172.30.0.14:10001->172.30.0.1:41364: i/o timeout"}
    2024-08-21 09:22:05.484 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "ddfcdf251ebc09ae48809e1b3fcf2566", "operationID": "1724227857243", "opUserID": "10346", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:22:06.748 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724227857243", "opUserID": "10346", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:22:34.215 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]                          readMessage                                         {"remoteAddr": "172.30.0.1:54520", "platform": "Android", "connID": "deb7a9b20e089938494065a68cd20cb9", "operationID": "1724231941476187150", "opUserID": "10326", "messageType": "-1", "error": "read tcp 172.30.0.14:10001->172.30.0.1:54520: i/o timeout"}
    2024-08-21 09:22:37.529 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "8433901f994d701f835435752b54cfeb", "operationID": "1724219829473", "opUserID": "10326", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:22:39.035 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724219829473", "opUserID": "10326", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:22:50.196 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724232167318", "opUserID": "10321", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:22:50.199 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "dd2ea2d4fd9de572e366f4aa5b00cd4f", "operationID": "1724232167318", "opUserID": "10321", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:07.549 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]                          readMessage                                         {"remoteAddr": "172.30.0.1:52492", "platform": "Android", "connID": "8433901f994d701f835435752b54cfeb", "operationID": "1724232154427024590", "opUserID": "10326", "messageType": "-1", "error": "read tcp 172.30.0.14:10001->172.30.0.1:52492: i/o timeout"}
    2024-08-21 09:23:09.812 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "6fa77d173bc130888f6bffcd6753d198", "operationID": "1724219829473", "opUserID": "10326", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:09.826 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724219829473", "opUserID": "10326", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:20.445 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]                          readMessage                                         {"remoteAddr": "172.30.0.1:42526", "platform": "Android", "connID": "ddfcdf251ebc09ae48809e1b3fcf2566", "operationID": "1724232125908237498", "opUserID": "10346", "messageType": "-1", "error": "read tcp 172.30.0.14:10001->172.30.0.1:42526: i/o timeout"}
    2024-08-21 09:23:21.858 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724232203475106492", "opUserID": "10366", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:21.936 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724232201274", "opUserID": "10366", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:22.517 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "8a29d4eb665146fe897ebb877e7f3264", "operationID": "1724227857243", "opUserID": "10346", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:22.673 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724232206227597910", "opUserID": "10366", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:22.700 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724232201444", "opUserID": "10366", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:23.610 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724227857243", "opUserID": "10346", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:29.156 ERROR   [PID:1621]      openim-api                  [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetConversationsHasReadAndMaxSeq    {"platform": "Android", "operationID": "1724232208244", "opUserID": "10366", "funcName": "/openim.msg.msg/GetConversationsHasReadAndMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:29.174 ERROR   [PID:1598]      openim-msggateway           [version:3.8.0]     [mw/rpc_client_interceptor.go:50]                   RPC Client Response Error - GetMaxSeq               {"platform": "Android", "connID": "bb022eb62ede520229a4522c26cb89aa", "operationID": "1724232209574825373", "opUserID": "10366", "funcName": "/openim.msg.msg/GetMaxSeq", "error": "rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 10.0.1.150:10130: connect: connection refused\"; last resolver error: produced zero addresses"}
    2024-08-21 09:23:52.518 WARN    [PID:1598]      openim-msggateway           [version:3.8.0]     [msggateway/client.go:140]

重启服务后，恢复正常。

Screenshots Link

No response

FGadvancer commented 3 weeks ago

Can you provide information use "docker images" for us to troubleshoot, thanky!!!

cipchk commented 3 weeks ago 
openim/openim-web-front                              release-v3.8.0                 b08f406b8c68   2 weeks ago         85.7MB
openim/openim-chat                                   release-v1.8                   5c3f739cfc42   2 weeks ago         465MB
openim/openim-server                                 release-v3.8                   07c845b2afda   2 weeks ago         985MB
openim/openim-admin-front                            release-v1.8.0                 2df59c51dcbf   2 weeks ago         48.1MB
grafana/grafana                                      11.0.1                         97bf66938f26   8 weeks ago         445MB
redis                                                latest                         9c893be668ac   3 months ago        116MB
prom/prometheus                                      v2.51.2                        051cb67876a6   4 months ago        262MB
quay.io/coreos/etcd                                  v3.5.13                        bef2e3b99162   4 months ago        57.8MB
prom/alertmanager                                    v0.27.0                        11f11916f8cd   5 months ago        70.2MB
minio/minio                                          RELEASE.2024-01-11T07-46-16Z   f5d82361e10f   7 months ago        151MB
bitnami/kafka                                        3.5.1                          f79889e4e4af   8 months ago        558MB
prom/node-exporter                                   v1.7.0                         72c9c2088986   9 months ago        22.7MB
mongo                                                6.0.2                          b70536aeb250   22 months ago       695MB
redis                                                7.0.0                          53aa81e8adfa   2 years ago         117MB