roachtest: tpcc/headroom/n4cpu16 failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.16k stars 3.82k forks source link

roachtest: tpcc/headroom/n4cpu16 failed #135802

Closed cockroach-teamcity closed 2 hours ago

cockroach-teamcity commented 2 days ago

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(monitor.go:149).Wait: monitor failure: full command output in run_090852.474952565_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

arch=amd64
cloud=aws
coverageBuild=false
cpu=16
encrypted=true
fs=ext4
localSSD=true
metamorphicLeases=epoch
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

/cc @cockroachdb/test-eng _{This test on roachdash | Improve this report!

Jira issue: CRDB-44721}

cockroach-teamcity commented 2 days ago

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(monitor.go:149).Wait: monitor failure: full command output in run_125339.853361564_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=16
encrypted=false
fs=zfs
localSSD=true
metamorphicLeases=expiration
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 day ago

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(monitor.go:149).Wait: monitor failure: full command output in run_090517.691577321_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

arch=amd64
cloud=aws
coverageBuild=false
cpu=16
encrypted=true
fs=ext4
localSSD=true
metamorphicLeases=leader
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 day ago

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(monitor.go:149).Wait: monitor failure: full command output in run_134747.454326116_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=16
encrypted=true
fs=ext4
localSSD=true
metamorphicLeases=default
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 7 hours ago

roachtest.tpcc/headroom/n4cpu16 failed with artifacts on master @ 5c5c9d6803d47848aa1960dd6642d5f2c1926814:

(monitor.go:149).Wait: monitor failure: full command output in run_091524.111041856_n4_cockroach-workload-r.log: invalid node selector '4', cluster contains 3 nodes
test artifacts and logs in: /artifacts/tpcc/headroom/n4cpu16/run_1

Parameters:

arch=amd64
cloud=aws
coverageBuild=false
cpu=16
encrypted=false
fs=ext4
localSSD=true
metamorphicLeases=default
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #135903 roachtest: tpcc/headroom/n4cpu16 failed [C-test-failure O-roachtest O-robot T-testeng branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

herkolategan commented 6 hours ago

  |   | <truncated> ... generator...
  |   | I241120 09:08:55.813438 1 workload/cli/run.go:515  [-] 3  creating load generator... done (took 2.309226875s)
  |   | E241120 09:12:28.854342 1 workload/cli/run.go:599  [-] 4  error in newOrder: update district failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" meta={id=b5cbfce1 key=/Table/107/1/672/3/0 iso=Serializable pri=0.03128564 epo=0 ts=1732093861.957584744,0 min=1732093861.957584744,0 seq=0} lock=true stat=PENDING rts=1732093861.957584744,0 wto=false gul=1732093862.457584744,0 (SQLSTATE 40001)
  |   | Error: error in newOrder: update district failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" meta={id=b5cbfce1 key=/Table/107/1/672/3/0 iso=Serializable pri=0.03128564 epo=0 ts=1732093861.957584744,0 min=1732093861.957584744,0 seq=0} lock=true stat=PENDING rts=1732093861.957584744,0 wto=false gul=1732093862.457584744,0 (SQLSTATE 40001)

herkolategan commented 2 hours ago

Fixed by: https://github.com/cockroachdb/cockroach/pull/135915