cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.17k stars 3.82k forks source link

roachtest: sysbench/oltp_point_select/nodes=3/cpu=32/conc=256 failed #119949

Closed cockroach-teamcity closed 8 months ago

cockroach-teamcity commented 8 months ago

roachtest.sysbench/oltp_point_select/nodes=3/cpu=32/conc=256 failed with artifacts on master @ bf013ea0a5311726e65d37e8f047ce39ea2d5f10:

(test_runner.go:1161).runTest: test timed out (3h0m0s)
test artifacts and logs in: /artifacts/sysbench/oltp_point_select/nodes=3/cpu=32/conc=256/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/test-eng

This test on roachdash | Improve this report!

Jira issue: CRDB-36408

renatolabs commented 8 months ago

Test timed out with errors like:

W240305 15:49:46.405412 159168 kv/kvclient/kvcoord/dist_sender.go:1952 ⋮ [T1,Vsystem,n1,job=‹NEW SCHEMA CHANGE id=948906474959110145›,intExec=‹write-job-info-delete›] 1119 slow range RPC: have been waiting 60.24s (60 attempts) for RPC DeleteRange [/Table/53/1/‹948906474959110145›/‹"legacy_payload"›,/Table/53/1/‹948906474959110145›/‹"legacy_payload"›/‹PrefixEnd›), [txn: 1455502a], [max_span_request_keys: 600], [target_bytes: 0] to r55:‹/Table/5{3-4}› [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=4]; resp: ‹failed to send RPC›: sending to all replicas failed; last error: invalid batch (LeaseInfo [/Table/53]): batch request missing store ID

The timing and failure mode makes it seem like this is a dupe of #119939. @erikgrinaker do you mind confirming this is the case?

erikgrinaker commented 8 months ago

Yep, dupe of #119939.