search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

roachtest: loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed #135808

Closed cockroach-teamcity closed 4 days ago

cockroach-teamcity commented 5 days ago

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(cluster.go:2456).Run: full command output in run_101734.077029937_n6_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-44727

cockroach-teamcity commented 5 days ago

roachtest.loqrecovery/half-online/workload=tpcc/rangeSize=16mb failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(cluster.go:2456).Run: full command output in run_114825.466476098_n6_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/loqrecovery/half-online/workload=tpcc/rangeSize=16mb/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

kvoli commented 5 days ago

Failure is from:

Error: error in newOrder: update stock failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: WriteTooOldError: write for key /Table/113/1/32/72257/0 at timestamp 1732103324.790649698,0 too old; must write at or above 1732103324.790649698,2: "sql txn" meta={id=0b884ee4 key=/Table/107/1/32/9/0 iso=Serializable pri=0.00115419 epo=0 ts=1732103324.790649698,2 min=1732103324.790649698,0 seq=4} lock=true stat=PENDING rts=1732103324.790649698,0 wto=false gul=1732103325.290649698,0 (SQLSTATE 40001)
run_114825.466476098_n6_cockroach-workload-r: 2024/11/20 11:48:45 cluster.go:2493: > result: COMMAND_PROBLEM: exit status 1

This looks similar to another test (https://github.com/cockroachdb/cockroach/issues/135787) which also just started failing. All this points to a recent change in the workload runner? cc @miraradeva this could be related to your recent commit that merged yesterday:

I'm going to mark this as a testing issue, since the workload runners could be considered part of the test.

blathers-crl[bot] commented 4 days ago

Based on the specified backports for linked PR #135915, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.