search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.01k stars 3.79k forks source link

roachtest: cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed #132284

Open cockroach-teamcity opened 4 days ago

cockroach-teamcity commented 4 days ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ fd4b1464dbd6e385c6e51af26fe294fd2023a259:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-42920

cockroach-teamcity commented 3 days ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:

(cluster.go:2449).Run: full command output in run_062209.635850484_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:

(cluster.go:2449).Run: full command output in run_082334.425856609_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

wenyihu6 commented 3 days ago

Similar signs as https://github.com/cockroachdb/cockroach/issues/132413. Splits didn't finish, slow span latches, slow replica rpc, slow heartbeats.

|   | Error: executing ALTER TABLE kv SPLIT AT VALUES (5509895122145396772): pq: replica unavailable: (n2,s2):4 unable to serve request to r25074:/Table/106/1/55{09341725357153370-10448518933640174} [(n5,s5):1, (n2,s2):4, (n1,s1):3, next=5, gen=64, sticky=9223372036.854775807,2147483647]: closed timestamp: 1728636708.180897336,0 (2024-10-11 08:51:48); raft status: {"id":"4","term":9,"vote":"4","commit":35,"lead":"4","raftState":"StateLeader","applied":35,"progress":{"1":{"match":0,"next":28,"state":"StateSnapshot"},"3":{"match":35,"next":60,"state":"StateReplicate"},"4":{"match":58,"next":60,"state":"StateReplicate"}},"leadtransferee":"0"}: have been waiting 62.00s for slow proposal HeartbeatTxn [/Local/Range/Table/106/1/5509341725357153370/RangeDescriptor], [txn: de3dfa86]

teamcity-17270244-1728625790-49-n7cpu16-0004> I241011 08:30:08.113355 453 kv/kvserver/store_remove_replica.go:152 ⋮ [T1,Vsystem,n4,s4,r29914/2:‹×›,raft] 19943 removing replica r29914/2
teamcity-17270244-1728625790-49-n7cpu16-0004> W241011 08:30:08.113722 842019 kv/kvclient/kvcoord/dist_sender.go:2746 ⋮ [T1,Vsystem,n4,s4,r1/3:‹×›] 19944 slow replica RPC: have been waiting 15.47s (1 attempts) for RPC PushTxn(PUSH_TIMESTAMP,00000000->31c64fd0) [/Local/Range/Table/106/1/‹×›/‹×›] to replica (n1,s1):1; resp: ‹×›
cockroach-teamcity commented 2 days ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(cluster.go:2449).Run: full command output in run_092613.385122981_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(cluster.go:2449).Run: full command output in run_063353.527584632_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 21 hours ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(cluster.go:2449).Run: full command output in run_062541.187973244_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 19 hours ago

roachtest.cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv0/nodes=5/cpu=16/ranges=100k/control/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

andrewbaptist commented 8 hours ago

This is caused by the recent pebble bump: 124e6c86c10. Reassigning to storage.