cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

roachtest: cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed #134506

Open cockroach-teamcity opened 3 weeks ago

cockroach-teamcity commented 3 weeks ago

roachtest.cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed with artifacts on release-24.3 @ 7252771de42bf0752052fae4f945c0e7a88d0ecd:

(cluster.go:2451).Run: full command output in run_133812.393161898_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #134139 roachtest: cdc/workload/kv100/nodes=5/cpu=16/ranges=100k/server=scheduler/protocol=mux/format=json/sink=null failed [A-cdc A-kv C-test-failure O-roachtest O-robot T-cdc T-kv branch-release-23.2.15-rc]

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-44121

rharding6373 commented 3 weeks ago

Range splits are slow:

  |   | I241107 13:45:47.247837 453 workload/workloadsql/workloadsql.go:194  [-] 60  finished 58000 of 100000 splits
  |   | I241107 13:45:55.074672 453 workload/workloadsql/workloadsql.go:194  [-] 61  finished 59000 of 100000 splits
  |   | I241107 13:46:02.869339 453 workload/workloadsql/workloadsql.go:194  [-] 62  finished 60000 of 100000 splits
  |   | I241107 13:46:10.698131 453 workload/workloadsql/workloadsql.go:194  [-] 63  finished 61000 of 100000 splits
  |   | I241107 13:46:18.528485 453 workload/workloadsql/workloadsql.go:194  [-] 64  finished 62000 of 100000 splits
  |   | I241107 13:46:26.314484 453 workload/workloadsql/workloadsql.go:194  [-] 65  finished 63000 of 100000 splits
  |   | I241107 13:46:34.122698 453 workload/workloadsql/workloadsql.go:194  [-] 66  finished 64000 of 100000 splits
  |   | Error: executing ALTER TABLE kv SPLIT AT VALUES (1499428597745462478): dial tcp 10.142.0.60:26257: connect: connection timed out

Removing the release blocker label.