search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

roachtest: cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed #132297

Closed cockroach-teamcity closed 4 weeks ago

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ fd4b1464dbd6e385c6e51af26fe294fd2023a259:

(cluster.go:2478).Run: full command output in run_085022.952770349_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-42932

rharding6373 commented 1 month ago

From run_085022.952770349_n7_cockroach-workload-i.log:

Error: executing ALTER TABLE kv SPLIT AT VALUES (1832573464267990482): pq: replica unavailable: (n3,s3):1 unable to serve request to r14246:/Table/106/1/1832{204533075828214-757929864071616} [(n3,s3):1, (n5,s5):4, (n2,s2):3, next=5, gen=69, sticky=9223372036.854775807,2147483647]: closed timestamp: 1728552173.402855460,0 (2024-10-10 09:22:53); raft status: {"id":"1","term":40,"vote":"1","commit":69,"lead":"4","raftState":"StateFollower","applied":63,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal HeartbeatTxn [/Local/Range/Table/106/1/1832204533075828214/RangeDescriptor], [txn: 2c11671e]
cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 645eb8c99796b3b88f5631aa0fc92a011010ce64:

(cluster.go:2449).Run: full command output in run_083023.040956125_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

wenyihu6 commented 1 month ago

Signs of nodes being overloaded:

raft ready handling: 0.70s [append=0.00s, apply=0.69s, , other=0.00s], wrote [apply=10 KiB (29 in 25 batches)], state_assertions=12; node might be overloaded

slow replica RPC: have been waiting 10.28s (0 attempts) for RPC AdminScatter [/Table/106/1/‹×›,/Table/106/1/‹×›/‹×›) to replica (n1,s1):1; resp: ‹×›

123 applied lease after ~10.01s replication lag, client traffic may have been delayed [lease=repl=(n1,s1):6 seq=5 start=1728635662.042788637,0 exp=1728635668.042714247,0 pro=1728635662.042714247,0 prev=repl=(n2,s2):4VOTER_INCOMING seq=4 start=1728635432.446735102,0 epo=1 min-exp=1728635438.446703383,0 pro=1728635432.448175523,0 acquisition-type=Transfer]
cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 30dbb173d0f083b35cf9eb8093832a5dd764c5af:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(cluster.go:2449).Run: full command output in run_075346.792866206_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ a0f39e7ac9574756063bc90bba6bc532b45c33d4:

(test_runner.go:1308).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 49ca24cedb042579e9645c206640d59975805d12:

(cluster.go:2449).Run: full command output in run_081245.625997888_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 49ca24cedb042579e9645c206640d59975805d12:

(cluster.go:2449).Run: full command output in run_095722.266455309_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

rharding6373 commented 1 month ago 
  | Wraps: (3) Node 7. Command with error:
  |   | ```
  |   | ./cockroach workload init kv --splits 100000 {pgurl:1}
  |   | ```
  |   | <truncated> ... rkloadsql/workloadsql.go:194  [-] 51  finished 49000 of 100000 splits
  |   | I241010 09:22:30.890028 464 workload/workloadsql/workloadsql.go:194  [-] 52  finished 50000 of 100000 splits
  |   | I241010 09:23:12.030787 464 workload/workloadsql/workloadsql.go:194  [-] 53  finished 51000 of 100000 splits
  |   | I241010 09:24:10.019556 464 workload/workloadsql/workloadsql.go:194  [-] 54  finished 52000 of 100000 splits
  |   | Error: executing ALTER TABLE kv SPLIT AT VALUES (1832573464267990482): pq: replica unavailable: (n3,s3):1 unable to serve request to r14246:/Table/106/1/1832{204533075828214-757929864071616} [(n3,s3):1, (n5,s5):4, (n2,s2):3, next=5, gen=69, sticky=9223372036.854775807,2147483647]: closed timestamp: 1728552173.402855460,0 (2024-10-10 09:22:53); raft status: {"id":"1","term":40,"vote":"1","commit":69,"lead":"4","raftState":"StateFollower","applied":63,"progress":{},"leadtransferee":"0"}: have been waiting 61.50s for slow proposal HeartbeatTxn [/Local/Range/Table/106/1/1832204533075828214/RangeDescriptor], [txn: 2c11671e]
cockroach-teamcity commented 4 weeks ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 5be5b0b52ff79b98689b2282a8b25cf9eb50ec40:

(cluster.go:2449).Run: full command output in run_082731.476118480_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on master @ 5be5b0b52ff79b98689b2282a8b25cf9eb50ec40:

(cluster.go:2449).Run: full command output in run_102353.297259341_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!