roachtest: restore/online/workload=true/tpcc/350GB/gce/inc-count=0/nodes=4/cpus=8 failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.16k stars 3.82k forks source link

roachtest: restore/online/workload=true/tpcc/350GB/gce/inc-count=0/nodes=4/cpus=8 failed #135813

Closed cockroach-teamcity closed 22 hours ago

cockroach-teamcity commented 2 days ago

roachtest.restore/online/workload=true/tpcc/350GB/gce/inc-count=0/nodes=4/cpus=8 failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(monitor.go:149).Wait: monitor failure: Workload context was not cancelled. Error returned by workload cmd: full command output in run_115601.305984353_n5_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/restore/online/workload=true/tpcc/350GB/gce/inc-count=0/nodes=4/cpus=8/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=8
encrypted=false
fs=ext4
localSSD=false
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery _{This test on roachdash | Improve this report!

Jira issue: CRDB-44732}

msbutler commented 1 day ago

Another workload flake:

E241120 11:56:03.586044 1 workload/cli/run.go:599  [-] 5  error in newOrder: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to encountered recently written committed value /Table/106/1/0/0 @1732103762.592056130,1): "sql txn" meta={id=3856e2fd key=/Table/107/1/0/9/0 iso=Serializable pri=0.04702080 epo=0 ts=1732103762.874366032,1 min=1732103762.381188969,0 seq=34} lock=true stat=PENDING rts=1732103762.381188969,0 wto=false gul=1732103762.881188969,0 (SQLSTATE 40001)
Error: error in newOrder: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionRetryError: retry txn (RETRY_SERIALIZABLE - failed preemptive refresh due to encountered recently written committed value /Table/106/1/0/0 @1732103762.592056130,1): "sql txn" meta={id=3856e2fd key=/Table/107/1/0/9/0 iso=Serializable pri=0.04702080 epo=0 ts=1732103762.874366032,1 min=1732103762.381188969,0 seq=34} lock=true stat=PENDING rts=1732103762.381188969,0 wto=false gul=1732103762.881188969,0 (SQLSTATE 40001)
run_115601.305984353_n5_cockroach-workload-r: 2024/11/20 11:56:03 cluster.go:2493: > result: COMMAND_PROBLEM: exit status 1

cockroach-teamcity commented 1 day ago

roachtest.restore/online/workload=true/tpcc/350GB/gce/inc-count=0/nodes=4/cpus=8 failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(monitor.go:149).Wait: monitor failure: Workload context was not cancelled. Error returned by workload cmd: full command output in run_130016.625620881_n5_cockroach-workload-r.log: COMMAND_PROBLEM: exit status 1
test artifacts and logs in: /artifacts/restore/online/workload=true/tpcc/350GB/gce/inc-count=0/nodes=4/cpus=8/run_1

Parameters:

arch=amd64
cloud=gce
coverageBuild=false
cpu=8
encrypted=false
fs=ext4
localSSD=false
runtimeAssertionsBuild=false
ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

blathers-crl[bot] commented 22 hours ago

Based on the specified backports for linked PR #135915, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}