search

databendlabs / databend

𝗗𝗮𝘁𝗮, 𝗔𝗻𝗮𝗹𝘆𝘁𝗶𝗰𝘀 & 𝗔𝗜. Modern alternative to Snowflake. Cost-effective and simple for massive-scale analytics. https://databend.com
https://docs.databend.com
Other
7.88k stars 751 forks source link

flaky test: can not forward any more: max number of forward reached #16749

Open everpcpc opened 3 weeks ago

everpcpc commented 3 weeks ago

3: statement failed: mysql client error: Server error: `ERROR HY000 (1105): MetaServiceError. Code: 2001, Text = can not forward any more: max number of forward reached.' [SQL] insert into t2 values(4,'b_8','c_8'); at tests/sqllogictests/suites/base/09_fuse_engine/./09_0036_merge_into_without_distributed_enable.test:74 at tests/sqllogictests/suites/base/09_fuse_engine/09_0026_merge_into.test:4

https://github.com/databendlabs/databend/actions/runs/11625478696/job/32376074474

everpcpc commented 3 weeks ago

cc @drmingdrmer could you please help with this?

drmingdrmer commented 2 weeks ago

It seems the CI machine was under a heavy load, node-1 the leader failed to send a message to node-3 and node-3 start to elect. During this period, there is no Leader and a request can not be forward.

meta-node-3 log:

2024-11-01T07:13:31.922424+00:00[UTC]  INFO openraft::core::raft_core: raft_core.rs:1540 election timeout passed, about to elect
2024-11-01T07:13:31.922481+00:00[UTC]  INFO openraft::core::raft_core: raft_core.rs:1550 do trigger election