search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.11k stars 3.81k forks source link

roachtest: ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed #128530

Closed cockroach-teamcity closed 2 months ago

cockroach-teamcity commented 3 months ago

roachtest.ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global failed with artifacts on release-24.1 @ 2f1e69fc552d6b3e3ca761d4a3c9962ff25d2555:

(cluster.go:2432).Run: full command output in run_065742.417439974_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/ycsb/E/nodes=3/cpu=32/mvcc-range-keys=global/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-41052

herkolategan commented 2 months ago

This looks like a network blip (node 2, 10.12.5.225, connection failed):

  |   | <truncated> ... 808 06:57:43.484777 1 ccl/workloadccl/fixture.go:315  [-] 2  starting import of 1 tables
  |   | I240808 06:57:48.878376 9 ccl/workloadccl/fixture.go:492  [-] 3  imported 1014 MiB in usertable table (1000000 rows, 0 index entries, took 5.38473375s, 188.34 MiB/s)
  |   | I240808 06:57:48.878482 1 ccl/workloadccl/fixture.go:323  [-] 4  imported 1014 MiB bytes in 1 tables (took 5.393596727s, 188.03 MiB/s)
  |   | I240808 06:57:48.903395 1 workload/workloadsql/workloadsql.go:148  [-] 5  starting 3 splits
  |   | I240808 06:57:49.347535 1 workload/cli/run.go:641  [-] 6  random seed: 12651021336037080552
  |   | I240808 06:57:49.347664 1 workload/cli/run.go:431  [-] 7  creating load generator...
  |   | I240808 06:57:50.249160 1 workload/cli/run.go:470  [-] 8  creating load generator... done (took 901.492693ms)
  |   | E240808 06:59:03.582940 1 workload/cli/run.go:573  [-] 9  write failed: write tcp 10.12.2.157:47688 -> 10.12.5.225:26257: write: connection reset by peer
  |   | Error: write failed: write tcp 10.12.2.157:47688->10.12.5.225:26257: write: connection reset by peer
  | Wraps: (4) COMMAND_PROBLEM
  | Wraps: (5) exit status 1

Doesn't seem to be anything suspicious in node 2's logs.

May be related to: https://github.com/cockroachdb/cockroach/issues/128145

DarrylWong commented 2 months ago

Yeah this run didn't get https://github.com/cockroachdb/cockroach/pull/128625 and appears to follow the same pattern as the others. We haven't seen this in a while so I think we can close it out.