search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.01k stars 3.79k forks source link

roachtest: lease-preferences/manual-violating-transfer failed #123998

Closed cockroach-teamcity closed 5 months ago

cockroach-teamcity commented 5 months ago

roachtest.lease-preferences/manual-violating-transfer failed with artifacts on master @ 4c2e7761acd050aaee565443932b6b0eca55620b:

(test_runner.go:1237).runTest: test timed out (30m0s)
test artifacts and logs in: /artifacts/lease-preferences/manual-violating-transfer/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-38653

nvanbenschoten commented 5 months ago

We see the test transfer leases away from the preferred zone at 10:32:45:

10:32:45 lease_preferences.go:77: transferring leases to node 5 and waiting for lease preference conformance

We then see lease transfers over the next minute:

I240512 10:32:56.116834 134690 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r313/1:‹/Table/106/1/39{71302…-89730…}›] 13954  transferring lease to s2 usage=[batches/s=0.0 request_cpu/s=0µs raft_cpu/s=0µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]
I240512 10:32:56.118355 134690 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r705/3:‹/Table/106/1/-87{4423…-2580…}›] 13955  transferring lease to s2 usage=[batches/s=0.0 request_cpu/s=0µs raft_cpu/s=0µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]
I240512 10:32:56.119643 134690 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r702/2:‹/Table/106/1/-57{5884…-4042…}›] 13956  transferring lease to s2 usage=[batches/s=0.0 request_cpu/s=0µs raft_cpu/s=0µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]
I240512 10:32:56.120688 134690 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r229/2:‹/Table/106/1/-50{2171…-0328…}›] 13957  transferring lease to s2 usage=[batches/s=0.0 request_cpu/s=0µs raft_cpu/s=0µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]
I240512 10:32:56.122001 134690 13@kv/kvserver/lease_queue.go:138 ⋮ [T1,Vsystem,n5,lease,s5,r439/1:‹/Table/106/1/66{06551…-24979…}›] 13958  err=can't transfer r439 lease violating preferences, no suitable target
I240512 10:32:56.122123 134690 13@kv/kvserver/lease_queue.go:143 â‹® [T1,Vsystem,n5,lease,s5,r

A few of these hit can't transfer r439 lease violating preferences, no suitable target errors, but those seem to be retried.

At the end, 997 leases transfer back, but 3 remain (r607, r165, r518). These then trickle over the next 15 minutes:

240512 10:33:47.341477 144304 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r46/4:‹/Table/4{3-4}›] 13964  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:33:49.596118 144427 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r66/4:‹/Table/6{4-5}›] 13965  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:35:40.091754 149072 13@kv/kvserver/allocator/plan/lease.go:91 ⋮ [T1,Vsystem,n5,lease,s5,r607/1:‹/Table/106/1/54{45567…-63995…}›] 13966  lease transfer needed transfer(preferences), enqueuing
I240512 10:35:40.091972 149045 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r607/1:‹/Table/106/1/54{45567…-63995…}›] 13967  transferring lease to s2 usage=[batches/s=0.0 request_cpu/s=2µs raft_cpu/s=7µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]
I240512 10:36:22.928571 150829 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r50/4:‹/Table/4{7-8}›] 13968  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:36:28.002169 151042 13@kv/kvserver/allocator/plan/lease.go:91 ⋮ [T1,Vsystem,n5,lease,s5,r165/1:‹/Table/106/1/16{12477…-30905…}›] 13969  lease transfer needed transfer(preferences), enqueuing
I240512 10:36:28.002335 150998 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r165/1:‹/Table/106/1/16{12477…-30905…}›] 13970  transferring lease to s2 usage=[batches/s=0.0 request_cpu/s=2µs raft_cpu/s=5µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]
I240512 10:37:08.574011 152688 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r17/4:‹/Table/1{4-5}›] 13971  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:38:00.961573 155013 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r7/4:‹/Table/{3-4}›] 13972  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:38:39.261765 156632 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r52/4:‹/Table/5{0-1}›] 13973  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:38:43.767165 156889 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r2/4:‹/System/NodeLiveness{-Max}›] 13974  no lease transfer needed no-transfer(no valid targets), not enqueueing
I240512 10:39:15.862536 158294 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r42/4:‹/Table/{39-40}›] 13975  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:39:45.133467 159500 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r22/4:‹/Table/{19-20}›] 13976  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:40:37.450481 161716 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r67/4:‹/Table/6{5-6}›] 13977  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:40:53.750843 162543 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r15/4:‹/Table/1{2-3}›] 13978  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:41:11.156499 163319 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r31/4:‹/Table/2{8-9}›] 13979  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:41:29.769833 164075 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r52/4:‹/Table/5{0-1}›] 13980  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:41:34.283110 164258 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r42/4:‹/Table/{39-40}›] 13981  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:42:06.430961 165591 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r50/4:‹/Table/4{7-8}›] 13982  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:42:53.800595 167532 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r36/4:‹/Table/3{3-4}›] 13983  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:43:50.754262 169866 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r26/5:‹/Table/2{3-4}›] 13984  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:44:25.707303 171289 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r31/4:‹/Table/2{8-9}›] 13985  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:44:41.495371 171990 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r7/4:‹/Table/{3-4}›] 13986  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:44:47.694553 172243 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r33/4:‹/NamespaceTable/{30-Max}›] 13987  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:45:16.440319 173401 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r2/4:‹/System/NodeLiveness{-Max}›] 13988  no lease transfer needed no-transfer(no valid targets), not enqueueing
I240512 10:45:33.350405 174103 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r17/4:‹/Table/1{4-5}›] 13989  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:45:57.026280 175081 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r66/4:‹/Table/6{4-5}›] 13990  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:46:32.528680 176555 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r15/4:‹/Table/1{2-3}›] 13991  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:46:46.053365 177098 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r67/4:‹/Table/6{5-6}›] 13992  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:47:46.339336 179643 13@kv/kvserver/allocator/plan/lease.go:96 ⋮ [T1,Vsystem,n5,lease,s5,r44/4:‹/Table/4{1-2}›] 13993  no lease transfer needed no-transfer(balanced), not enqueueing
I240512 10:48:02.149325 180311 13@kv/kvserver/allocator/plan/lease.go:91 ⋮ [T1,Vsystem,n5,lease,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›] 13994  lease transfer needed transfer(preferences), enqueuing
I240512 10:48:02.149521 180291 13@kv/kvserver/lease_queue.go:143 ⋮ [T1,Vsystem,n5,lease,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›] 13995  transferring lease to s1 usage=[batches/s=0.0 request_cpu/s=0µs raft_cpu/s=1µs write(keys)/s=0.0 write(bytes)/s=0 B read(keys)/s=0.0 read(bytes)/s=0 B]

This should have failed after 5 minutes, but the test's timeout was broken (see https://github.com/cockroachdb/cockroach/pull/124034), so instead, the test harness fires after 30 minutes, just before the last lease is transferred over. I'd guess that this will fail more often once that PR merges.

So it looks like the reactive transfers from https://github.com/cockroachdb/cockroach/commit/ba13d19481da623d96708a86a9459b5e64e65494 aren't working for these 3 leases. This is similar to what we saw in https://github.com/cockroachdb/cockroach/issues/123866, though I closed that with some suspicion that Azure was to blame.

@kvoli since you recently touched this test and this area, can I ask you to take a closer look at why these three ranges weren't eagerly enqueued? Is it surprising that we don't see this log line fire?

kvoli commented 5 months ago

So it looks like the reactive transfers from https://github.com/cockroachdb/cockroach/commit/ba13d19481da623d96708a86a9459b5e64e65494 aren't working for these 3 leases. This is similar to what we saw in https://github.com/cockroachdb/cockroach/issues/123866, though I closed that with some suspicion that Azure was to blame.

None of the leases should be less-preferred, only violating the preference from being on n5 (dc=3), so https://github.com/cockroachdb/cockroach/commit/ba13d19481da623d96708a86a9459b5e64e65494 won't apply.

https://github.com/cockroachdb/cockroach/blob/a3cde79a723a0e908cbf08737f4e5a648feef58f/pkg/cmd/roachtest/tests/lease_preferences.go#L158-L162

can I ask you to take a closer look at why these three ranges weren't eagerly enqueued? Is it surprising that we don't see this log line fire?

Will do, I'm surprised as well. The log line is firing when looking at these logs:

r607

``` I240512 10:32:55.653744 224 kv/kvserver/replica_proposal.go:560 ⋮ [T1,Vsystem,n5,s5,r607/1:‹/Table/106/1/54{45567…-63995…}›,raft] 43012 acquired lease violates lease preferences, enqueuing for transfer [lease=repl=(n5,s5):1 seq=8 start=1715509975.652418247,0 exp=1715509981.652374547,0 pro=1715509975.652374547,0 preferences=[{[‹+dc=1›]} {[‹+dc=2›]}]] ```

r165

``` I240512 10:32:55.296963 233 kv/kvserver/replica_proposal.go:560 ⋮ [T1,Vsystem,n5,s5,r165/1:‹/Table/106/1/16{12477…-30905…}›,raft] 41525 acquired lease violates lease preferences, enqueuing for transfer [lease=repl=(n5,s5):1 seq=8 start=1715509975.295690176,0 exp=1715509981.295654521,0 pro=1715509975.295654521,0 preferences=[{[‹+dc=1›]} {[‹+dc=2›]}]] ```

r518

``` I240512 10:32:55.634014 212 kv/kvserver/replica_proposal.go:560 ⋮ [T1,Vsystem,n5,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›,raft] 42926 acquired lease violates lease preferences, enqueuing for transfer [lease=repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0 preferences=[{[‹+dc=1›]} {[‹+dc=2›]}]] ```

The enqueue fails with a LeaseRejectedError and isn't retried because it doesn't have the purgatory marker. This explains why these three didn't transfer, or get retried from purgatory. TBD why the lease queue processing is returning the LeaseRejectedError:

E240512 10:32:55.637160 140508 kv/kvserver/queue.go:1202 ⋮ [T1,Vsystem,n5,lease,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›] 42938  cannot replace lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 epo=1 pro=1715509975.633970829,0 with repl=(n5,s5):1 seq=0 start=1715509975.636799733,0 epo=1 pro=1715509975.636799733,0: ‹expected previous lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0, found repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 epo=1 pro=1715509975.633970829,0›

From https://github.com/cockroachdb/cockroach/blob/a3cde79a723a0e908cbf08737f4e5a648feef58f/pkg/kv/kvserver/batcheval/cmd_lease_request.go#L75-L78

kvoli commented 5 months ago

This is likely a race between the expiration to epoch lease upgrade here:

https://github.com/cockroachdb/cockroach/blob/a3cde79a723a0e908cbf08737f4e5a648feef58f/pkg/kv/kvserver/replica_proposal.go#L495-L495

and replicaCanBeProcessed, called by the queue before processing a replica:

https://github.com/cockroachdb/cockroach/blob/a3cde79a723a0e908cbf08737f4e5a648feef58f/pkg/kv/kvserver/queue.go#L1074-L1074

This lines up with the other logging we see, using r518 as an example:

# n5 first acquires lease after manual transfer
I240512 10:32:55.633877 212 kv/kvserver/replica_proposal.go:384 ⋮ [T1,Vsystem,n5,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›,raft] 42924  new range lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0 following repl=(n2,s2):4 seq=5 start=1715509932.464289154,0 epo=1 pro=1715509932.466375520,0
# expiration to epoch lease upgrade starts within leasePostApply (same function as above)
I240512 10:32:55.633979 212 kv/kvserver/replica_proposal.go:488 ⋮ [T1,Vsystem,n5,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›,raft] 42925  upgrading expiration lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0 to an epoch-based one
# replica is enqueued for transfer async
I240512 10:32:55.634014 212 kv/kvserver/replica_proposal.go:560 ⋮ [T1,Vsystem,n5,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›,raft] 42926  acquired lease violates lease preferences, enqueuing for transfer [lease=repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0 preferences=[{[‹+dc=1›]} {[‹+dc=2›]}]]
# expiration lease is upgraded to an epoch lease
I240512 10:32:55.636812 216 kv/kvserver/replica_proposal.go:384 ⋮ [T1,Vsystem,n5,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›,raft] 42936  new range lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 epo=1 pro=1715509975.633970829,0 following repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0
# replica process attempt, failing the pre-process check
E240512 10:32:55.637160 140508 kv/kvserver/queue.go:1202 ⋮ [T1,Vsystem,n5,lease,s5,r518/1:‹/Table/106/1/52{24427…-42855…}›] 42938  cannot replace lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 epo=1 pro=1715509975.633970829,0 with repl=(n5,s5):1 seq=0 start=1715509975.636799733,0 epo=1 pro=1715509975.636799733,0 : ‹expected previous lease repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 exp=1715509981.632774257,0 pro=1715509975.632774257,0, found repl=(n5,s5):1 seq=6 start=1715509975.632817914,0 epo=1 pro=1715509975.633970829,0›
kvoli commented 5 months ago

Removing A-testing, this is a bug.

cockroach-teamcity commented 5 months ago

roachtest.lease-preferences/manual-violating-transfer failed with artifacts on master @ b185402d34768c898d63a61de3e585e4723fbe81:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:225
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:280
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:157
                                main/pkg/cmd/roachtest/test_runner.go:1208
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    timed out before lease preferences satisfied
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForLeasePreferences
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:375
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runLeasePreferences.func2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:223
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runLeasePreferences
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:280
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerLeasePreferences.func3
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:157
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1208
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) timed out before lease preferences satisfied
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           lease-preferences/manual-violating-transfer
    Messages:       violating(true) duration=300.37s: [n1: 0, n2: 0, n3: 0, n4: 0, n5: 2] less-preferred(false) duration=0.00s: [n1: 0, n2: 0, n3: 0, n4: 0, n5: 0]
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/lease-preferences/manual-violating-transfer/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

roachtest.lease-preferences/manual-violating-transfer failed with artifacts on master @ 6300c3c3367ad46ac48bf24915cf0d73cae446a0:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:225
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:280
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:157
                                main/pkg/cmd/roachtest/test_runner.go:1214
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    timed out before lease preferences satisfied
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.waitForLeasePreferences
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:375
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runLeasePreferences.func2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:223
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runLeasePreferences
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:280
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerLeasePreferences.func3
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/lease_preferences.go:157
                      | main.(*testRunner).runTest.func2
                      |     main/pkg/cmd/roachtest/test_runner.go:1214
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) timed out before lease preferences satisfied
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           lease-preferences/manual-violating-transfer
    Messages:       violating(true) duration=299.88s: [n1: 0, n2: 0, n3: 0, n4: 0, n5: 4] less-preferred(false) duration=0.00s: [n1: 0, n2: 0, n3: 0, n4: 0, n5: 0]
(require.go:1360).NoError: FailNow called
test artifacts and logs in: /artifacts/lease-preferences/manual-violating-transfer/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!