cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.16k stars 3.82k forks source link

roachtest: alterpk-tpcc-500 failed #135787

Closed cockroach-teamcity closed 22 hours ago

cockroach-teamcity commented 2 days ago

roachtest.alterpk-tpcc-500 failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(cluster.go:2456).Run: full command output in run_065832.460188092_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/alterpk-tpcc-500/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-44707

rafiss commented 1 day ago

This had errors during the TPCC workload

E241120 07:01:46.886315 1 workload/cli/run.go:599  [-] 4  error in newOrder: update district failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" meta={id=3d5851f0 key=/Table/107/1/455/6/0 iso=Serializable pri=0.03259317 epo=0 ts=1732085945.267172477,0 min=1732085945.267172477,0 seq=0} lock=true stat=PENDING rts=1732085945.267172477,0 wto=false gul=1732085945.767172477,0 (SQLSTATE 40001)
Error: error in newOrder: update district failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" meta={id=3d5851f0 key=/Table/107/1/455/6/0 iso=Serializable pri=0.03259317 epo=0 ts=1732085945.267172477,0 min=1732085945.267172477,0 seq=0} lock=true stat=PENDING rts=1732085945.267172477,0 wto=false gul=1732085945.767172477,0 (SQLSTATE 40001)

This is unexpected to me; since these 40001 errors are supposed to be retried: https://github.com/cockroachdb/cockroach/blob/08c28dc2707305db2d3fadbec373425c9cdf3b58/pkg/workload/tpcc/tpcc.go#L1107-L1128

cockroach-teamcity commented 1 day ago

roachtest.alterpk-tpcc-500 failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(cluster.go:2456).Run: full command output in run_065326.070481496_n4_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
(alterpk.go:146).2: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/alterpk-tpcc-500/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #135867 roachtest: alterpk-tpcc-500 failed [B-runtime-assertions-enabled C-test-failure O-roachtest O-robot T-sql-foundations branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

blathers-crl[bot] commented 22 hours ago

Based on the specified backports for linked PR #135915, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.