[Bug]: sysbench test reported error FATAL: Worker threads failed to initialize within 30 seconds!

heni02 commented 3 months ago

Is there an existing issue for the same bug?

[X] #17382

Branch Name

main

Commit ID

6a43a6e7a

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9567035983/job/26400260798

企业微信截图_d7a65ef7-a6e7-42e3-b87e-ae6716c720ba

mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22kbI%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240618%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221718774634000%22,%22to%22:%221718774664000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke regression sysbench1000w nopk-load and nopk-point-select test

Additional information

No response

reusee commented 3 months ago

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22aaX%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bmatrixorigin_io_namespace%3D%5C%22mo-error-case--nightly-57a9e8776-20240624%5C%22%7D%20%7C%3D%20%60slow%20event%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-12h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

没有记录到慢的 I/O，不是 io 开销过大的问题。

reusee commented 3 months ago

原始错误信息

企业微信截图_a339aa93-a068-4a68-9016-dc1be1b8adfc

reusee commented 3 months ago

在 2024-06-24 04:00 左右的错误信息：https://grafana.ci.matrixorigin.cn/goto/KtEhzSQIR?orgId=1

可以看到有大量的 failed to init stats info for table 以及 connect: cannot assign requested address。后者似乎是因为fd耗尽，无法bind导致，可能是有网络连接未关闭，或者同时发起太多。

而且只有 04:00 这个时间有 connect: cannot assign requested address 的错误信息：https://grafana.ci.matrixorigin.cn/goto/Agb4iSwIR?orgId=1

reusee commented 3 months ago

Is there an existing issue for the same bug?
* [x]  I have checked the existing issues.
Branch Name

main

Commit ID

6a43a6e

Other Environment Information
- Hardware parameters:
- OS type:
- Others:
Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9567035983/job/26400260798

mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22kbI%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240618%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221718774634000%22,%22to%22:%221718774664000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce
tke regression sysbench1000w nopk-load and nopk-point-select test
Additional information

No response