search

matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.76k stars 274 forks source link

[Bug]: [date 6.2]tke regression: sysbench load data reported error ExpectedEOB #16598

Open heni02 opened 3 months ago

heni02 commented 3 months ago

Is there an existing issue for the same bug?

Branch Name

main

Commit ID

b9d103d8d

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9339030269/job/25715192203 现象描述：sysbench load data是同时10个并发load到10个1000w的表 第一次出错时间

企业微信截图_d05664ed-f814-4a75-bbb2-faa4843e1839

第二次出错时间

企业微信截图_5dc1a23a-f70d-4809-b9b9-9da882a9d666

两次报错从load开始到报错时间不到1分钟 tke 环境无节点重启

企业微信截图_253cb1fd-7606-4c2b-b22b-f55484f35da8

log： https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22_0q%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240602%5C%22%7D%20%7C%3D%20%60ExpectedEOB%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221717393909782%22,%22to%22:%221717404709782%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke regression sysbench 1000w random_points 100threads test

Additional information

No response

heni02 commented 3 months ago

@badboynt1 麻烦先确认是否和load性能变慢的pr有关

sukki37 commented 3 months ago

The issue observed is that the connection to 165002 encountered an error at 07:07:28 when the proxy reported that the backend CN (10.143.5.2:6001) was unreachable. Subsequently, the proxy redirected the connection to another CN (10.143.5.4), where it encountered an "ExpectedEOB" error.

image
badboynt1 commented 3 months ago

load性能回退的bug已经解决。 应该跟这个问题没有关系

volgariver6 commented 3 months ago

The issue observed is that the connection to 165002 encountered an error at 07:07:28 when the proxy reported that the backend CN (10.143.5.2:6001) was unreachable. Subsequently, the proxy redirected the connection to another CN (10.143.5.4), where it encountered an "ExpectedEOB" error.

image

https://grafana.ci.matrixorigin.cn/goto/eO0I4-ySR?orgId=1

07:07:28 这个时间发送的错误是刚建立连接的时候，第一个cn建立连接失败，然后又找到了第二个cn建立连接成功，开始执行sql，在执行的过程中发生了 ExpectedEOB 错误

volgariver6 commented 3 months ago

@reusee please help look into it

reusee commented 3 months ago

not working today.

reusee commented 3 months ago

无进展

reusee commented 3 months ago

无进展

reusee commented 3 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

无进展

reusee commented 2 months ago

working on other issues.

reusee commented 1 month ago

working on other issues.

reusee commented 1 month ago

working on other issues.

reusee commented 1 month ago

working on other issues.

reusee commented 1 month ago

working on other issues.

reusee commented 1 month ago

无进展

reusee commented 1 month ago

无进展

reusee commented 4 weeks ago

无进展

reusee commented 3 weeks ago

无进展

reusee commented 2 weeks ago

working on other issues.

reusee commented 2 weeks ago

working on other issues.

reusee commented 1 week ago

无进展

reusee commented 4 days ago

无进展