search

Tencent / Tendis

Tendis is a high-performance distributed storage system fully compatible with the Redis protocol.
http://tendis.cn
Other
2.87k stars 317 forks source link

集群出现大量shutdown socket failed.shutdown: Transport endpoint is not connected id日志 #274

Open epolar opened 1 week ago

epolar commented 1 week ago

Description

使用的 6 主集群压测，发现性能不是很稳定，从节点的 CPU 不会稳定在一个值附近，测试机和主节点相对比较稳定。

从节点: image

主节点: image

查看主节点看到有很多这样的日志:

I0626 14:17:10.784807 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583242
W0626 14:17:10.786993 132945 network.cpp:580] setResponse _isEnded, id:235581835 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.789252 132955 network.cpp:580] setResponse _isEnded, id:235582703 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.790391 132956 network.cpp:580] setResponse _isEnded, id:235583507 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.790848 132951 network.cpp:580] setResponse _isEnded, id:235583587 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.794304 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.794345 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583417
W0626 14:17:10.799111 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.799168 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235582423
W0626 14:17:10.808280 132948 network.cpp:580] setResponse _isEnded, id:235583600 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.812263 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.812304 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583128
W0626 14:17:10.823552 132950 network.cpp:580] setResponse _isEnded, id:235583592 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.844386 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.844444 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583362
I0626 14:17:10.850581 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583379
W0626 14:17:10.854308 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.854353 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583568
W0626 14:17:10.879565 132959 network.cpp:580] setResponse _isEnded, id:235583524 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.898736 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.898797 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583617
W0626 14:17:10.901582 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.901645 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583458
W0626 14:17:10.911201 132948 network.cpp:580] setResponse _isEnded, id:235582864 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.928401 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.928464 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583066
W0626 14:17:10.935529 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.935585 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583576
W0626 14:17:10.937903 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.937949 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583619
W0626 14:17:10.940001 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.940040 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235582764
W0626 14:17:10.959430 132963 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.959478 132963 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583620
W0626 14:17:10.965763 132962 network.cpp:1052] drainRspCallback:Broken pipe
I0626 14:17:10.965811 132962 network.cpp:1112] shutdown socket failed.shutdown: Transport endpoint is not connected id:235583022
W0626 14:17:10.969278 132952 network.cpp:580] setResponse _isEnded, id:235583080 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.969380 132952 network.cpp:580] setResponse _isEnded, id:235583431 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.969821 132952 network.cpp:580] setResponse _isEnded, id:235583213 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:10.977566 132949 network.cpp:580] setResponse _isEnded, id:235582823 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:17:11.038833 132963 network.cpp:1052] drainRspCallback:Broken pipe

同样的日志在从节点也有:

W0626 14:30:15.104251  9455 network.cpp:580] setResponse _isEnded, id:221840542 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.105410  9464 network.cpp:580] setResponse _isEnded, id:221840559 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.105499  9464 network.cpp:580] setResponse _isEnded, id:221840567 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.105628  9464 network.cpp:580] setResponse _isEnded, id:221840390 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.107861  9452 network.cpp:580] setResponse _isEnded, id:221840187 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.107964  9452 network.cpp:580] setResponse _isEnded, id:221836457 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.108062  9452 network.cpp:580] setResponse _isEnded, id:221840493 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.112968  9453 network.cpp:580] setResponse _isEnded, id:221840478 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.113885  9466 network.cpp:580] setResponse _isEnded, id:221840430 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.113974  9466 network.cpp:580] setResponse _isEnded, id:221840604 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114216  9466 network.cpp:580] setResponse _isEnded, id:221840344 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114296  9466 network.cpp:580] setResponse _isEnded, id:221837082 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114401  9452 network.cpp:580] setResponse _isEnded, id:221840391 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.114943  9452 network.cpp:580] setResponse _isEnded, id:221840489 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.115025  9452 network.cpp:580] setResponse _isEnded, id:221836942 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.115100  9452 network.cpp:580] setResponse _isEnded, id:221840615 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.115185  9452 network.cpp:580] setResponse _isEnded, id:221834083 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122390  9452 network.cpp:580] setResponse _isEnded, id:221831828 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122516  9452 network.cpp:580] setResponse _isEnded, id:221840579 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122613  9452 network.cpp:580] setResponse _isEnded, id:221837026 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.122516  9467 network.cpp:580] setResponse _isEnded, id:221840281 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.125306  9455 network.cpp:580] setResponse _isEnded, id:221836982 addr:remote_endpoint: Transport endpoint is not connected
W0626 14:30:15.130019  9455 network.cpp:580] setResponse _isEnded, id:221832154 addr:remote_endpoint: Transport endpoint is not connected

Your Environment

netiothreadnum 2 executorThreadNum 16

executorWorkPoolSize 3

rocks.max_background_jobs 6

allow-cross-slot true

checkkeytypeforsetcmd false noexpire true

rocks.disable_wal true