roachtest: tpcc/multiregion/survive=region/chaos=true failed

[cockroach-teamcity]

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on master @ 8eeb7f2ae3b2cede564b46ca47e2353fd147c061:

(tpcc.go:347).runTPCC: error detected during DRT: error at from 2024-11-20T13:05:36Z, to 2024-11-20T13:09:17Z on metric workload_tpcc_newOrder_error_total{instance="34.139.154.29:2111"}: expected errors, found from 266.000000, to 266.000000
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• metamorphicLeases=default
• runtimeAssertionsBuild=false
• ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-44740

[cockroach-teamcity]

Note: This build has runtime assertions enabled. If the same failure was hit in a run without assertions enabled, there should be a similar failure without this message. If there isn't one, then this failure is likely due to an assertion violation or (assertion) timeout.

roachtest.tpcc/multiregion/survive=region/chaos=true failed with artifacts on master @ eb2d2e19eb29d2747d9e267bd0612a69d066adad:

(tpcc.go:351).runTPCC: error detected during DRT: error at from 2024-11-21T13:46:32Z, to 2024-11-21T13:50:13Z on metric workload_tpcc_newOrder_error_total{instance="34.75.80.177:2111"}: expected errors, found from 166.000000, to 166.000000
test artifacts and logs in: /artifacts/tpcc/multiregion/survive=region/chaos=true/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=true
• fs=ext4
• localSSD=true
• metamorphicLeases=default
• runtimeAssertionsBuild=true
• ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[rafiss]

it appears that retries are broken, similar to https://github.com/cockroachdb/cockroach/issues/135787

E241121 13:35:06.118873 1 workload/cli/run.go:594  [-] 4  error in newOrder: update district failed: ERROR: restart transaction: TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_CLIENT_REJECT): "sql txn" meta={id=723cc390 key=/Table/109/1/"\x80"/6/9/0 iso=Serializable pri=0.02168515 epo=0 ts=1732196094.973506232,0 min=1732196094.973506232,0 seq=0} lock=true stat=PENDING rts=1732196094.973506232,0 wto=false gul=1732196095.473506232,0 (SQLSTATE 40001)

[blathers-crl[bot]]

Based on the specified backports for linked PR #135915, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}