search

Tencent / Tendis

Tendis is a high-performance distributed storage system fully compatible with the Redis protocol.
http://tendis.cn
Other
2.87k stars 317 forks source link

2.5.0-rocksdb-v6.23.3 存在大量得error告警 #207

Open erwadba opened 1 year ago

erwadba commented 1 year ago

Description

err日志中存在大量告警。暂时不清楚这个告警对tendis得影响。

[root@xx-xx-xx-xx log]# more tendisplus.xx-xx-xx-xx.root.log.ERROR.20220803-092706.5527
Log file created at: 2022/08/03 09:27:06
Running on machine: xx.xx.xx.xx
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0803 09:27:06.157007  5918 cluster_manager.cpp:4793] Cluster state changed: ok
E0803 09:27:06.276286  5917 cluster_manager.cpp:5408] IP address for this node updated to xx.xx.xx.xx
E0902 19:29:06.758924  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 19:29:06.758992  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 19:29:06.759140  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 19:56:28.314909  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 19:56:29.022790  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 19:56:40.656702  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 19:58:34.522926  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:01:20.157943  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:03:46.378731  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:06:22.161800  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:06:22.343012  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:06:25.996968  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:13:53.277976  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:16:09.008278  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:16:14.134089  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:18:30.316573  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:19:58.566766  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:22:25.230005  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:22:35.116926  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:25:57.335562  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:28:39.734496  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:29:49.264446  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:36:29.361735  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:39:03.049813  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:40:23.246389  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:40:23.669319  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:40:27.571303  5887 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected
E0902 20:40:27.744642  5886 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected

Expected Behavior

2.4.3-rocksdb-v5.13.4这个版本，没有类似的告警。怎么知道这个错误，是由什么引起的？

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Context

Your Environment

raffertyyu commented 1 year ago

能描述一下复现的步骤吗。方便定位问题。

erwadba commented 1 year ago

程序跑着跑着，就出现这个告警了。不知道怎么去模拟。
E0920 14:06:54.198989 6055 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected E0920 14:06:54.198992 6054 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected E0920 14:06:57.767958 6041 expire.cpp:171] getDbWithKeyLock failed, keySZ_xxxx err:-MOVED 7606 xx.xx.xx.xx:8903 E0920 14:06:57.794759 6042 expire.cpp:171] getDbWithKeyLock failed, keySZ_xxxx err:-MOVED 7606 xx.xx.xx.xx:8903 E0920 14:06:58.040333 6041 expire.cpp:171] getDbWithKeyLock failed, keySH_xxxx err:-MOVED 7377 xx.xx.xx.xx:8903 E0920 14:06:58.299054 6042 expire.cpp:171] getDbWithKeyLock failed, keySZ_xxxx err:-MOVED 6196 xx.xx.xx.xx:8903 E0920 14:06:58.348944 6042 expire.cpp:171] getDbWithKeyLock failed, keySZ_xxxx err:-MOVED 6196 xx.xx.xx.xx:8903 E0920 14:11:26.601855 6055 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected E0920 14:11:31.936156 6054 network.cpp:1080] shutdown socket failed.shutdown: Transport endpoint is not connected

raffertyyu commented 1 year ago

你配置文件里的bind或者port信息填写的是正确的吗。方便截个图吗。或者能否加下微信我跟进一下问题。github同名。

lyc0221 commented 11 months ago

我也遇到同样问题了，有进展吗？

image

erwadba commented 11 months ago

降级为低版本，就没有问题了 2.4.3-rocksdb-v5.13.4这个版本。用着挺稳定的

takenliu commented 9 months ago

这个日志不影响，属于正常现象，只是日志打印级别设置高了，后面的版本会取消该日志。 出现的场景： 如果连接的对方关闭了该连接，服务器方就会识别出"Connection reset by peer"，然后在shutdown的时候就会打印这个"shutdown: Transport endpoint is not connected"日志。

takenliu commented 9 months ago

@erwadba @lyc0221 除了打印这个日志，还有其他问题吗？

smile689 commented 6 months ago

@takenliu 您好，我们最近用tendisplus-2.5.0-rocksdb-v6.23.3.tgz， tendisplus-2.6.0-rocksdb-v6.23.3.tgz这些比较新的版本，还是会有这个日志打印出来。是不是tendis服务端自己会主动关闭连接，有可能客户端已经把这个连接关闭了。所以会有这个报错？会对虚拟机句柄或者socket连接数造成影响吗？