matrixorigin / matrixone

Hyperconverged cloud-edge native database
https://docs.matrixorigin.cn/en
Apache License 2.0
1.77k stars 274 forks source link

[Bug]: TPCH 5T lineitem table load reported internal error: message body 367389134 is too large, max is 104857600 #17541

Open heni02 opened 2 months ago

heni02 commented 2 months ago

Is there an existing issue for the same bug?

Branch Name

1.2-dev

Commit ID

74f04ebd0334312ef17f364fb498daa61585c582

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

回归测试tke环境(3cn)load tpch5T tables, lineitem表报错message body 367389134 is too large, max is 104857600 job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9933590939/job/27436928546

企业微信截图_f9060056-a084-4e6f-8a87-c3a306fc663c

lineitem表预计300亿行

mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22CG5%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-ben-nightly-74f04ebd0-20240715%5C%22%7D%20%7C%3D%20%60message%20body%20367389134%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221721034449526%22,%22to%22:%221721034756941%22%7D%7D%7D&schemaVersion=1&orgId=1

profile: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%226YZ%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%5C%22mo-ben-nightly-74f04ebd0-20240715%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22profileTypeId%22:%22memory:alloc_objects:count:space:bytes%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D%7D%5D,%22range%22:%7B%22from%22:%221721020835068%22,%22to%22:%221721041525594%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke tpch5T test

Additional information

No response

ouyuanning commented 2 months ago

image

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22CG5%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-ben-nightly-74f04ebd0-20240715%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221721034596000%22,%22to%22:%221721034598000%22%7D%7D%7D&schemaVersion=1&orgId=1

单个事务的rpc请求太大

volgariver6 commented 2 months ago

还没看,明天看一下这个问题

heni02 commented 2 months ago

[cn.rpc] max-message-size = "1000M" 配置调整到1000M,加载4h20min后报错ERROR 1105 (HY000) at line 1: write tcp4 10.142.194.74:37312->10.143.148.136:41010: write: connection reset by peer

企业微信截图_6b04a254-502c-429d-9233-ad1e12e1387d 企业微信截图_057cf474-9a93-4dc3-8a59-f5087e352159

集群环境没有重启

企业微信截图_b778a8cc-e592-40cc-86f1-82ea3cb0f353

mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22n99%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-ben-nightly-88e486e11-20240718%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221721322470954%22,%22to%22:%221721341803367%22%7D%7D%7D&schemaVersion=1&orgId=1

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9989751236/job/27612225587

volgariver6 commented 2 months ago

修改参数之后可以正常导入了,先改成s1级别