ccl/kvccl/kvfollowerreadsccl: TestFollowerReadsWithStaleDescriptor failed

[cockroach-teamcity]

ccl/kvccl/kvfollowerreadsccl.TestFollowerReadsWithStaleDescriptor failed on release-24.1 @ c4641acfdb8533ccf951535f408ec61df8437b62:

=== RUN   TestFollowerReadsWithStaleDescriptor
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestFollowerReadsWithStaleDescriptor691115469
    test_log_scope.go:81: use -show-logs to present logs inline

pkg/ccl/kvccl/kvfollowerreadsccl/followerreads_test.go:891: (TestFollowerReadsWithStaleDescriptor)
    NOTICE: .DistSenderI() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DistSenderI() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    followerreads_test.go:895: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/followerreads_test.go:895
            Error:          Not equal: 
                            expected: 1
                            actual  : 2
            Test:           TestFollowerReadsWithStaleDescriptor
    panic.go:626: -- test log scope end --
test logs left over in: outputs.zip/logTestFollowerReadsWithStaleDescriptor691115469
--- FAIL: TestFollowerReadsWithStaleDescriptor (91.21s)

Parameters:

• attempt=1
• race=true
• run=3
• shard=1
  Help

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-39653

[arulajmani]

Test failed during setup because the lease wasn't in the right place. Looks like a load based lease transfer:

I240619 10:09:56.884336 3455 13@kv/kvserver/store_rebalancer.go:501 ⋮ [T1,Vsystem,n1,s1,store-rebalancer,obj=‹cpu›] 7  considering load-based lease transfers for s1 with (queries-per-second=32.8 cpu-per-second=517ms) load, mean=(queries-per-second=8.2 cpu-per-second=130ms), upperThreshold=(queries-per-second=8.2 cpu-per-second=180ms)
I240619 10:09:56.886302 3455 13@kv/kvserver/store_rebalancer.go:865 ⋮ [T1,Vsystem,n1,s1,store-rebalancer,obj=‹cpu›] 8  transferring lease for r68 load=(queries-per-second=1.1 cpu-per-second=3ms) to store s2 load=(queries-per-second=0.0 cpu-per-second=1ms) from local store s1 load=(queries-per-second=32.8 cpu-per-second=517ms)
I240619 10:09:56.886648 3455 13@kv/kvserver/replicate_queue.go:1009 ⋮ [T1,Vsystem,n1,s1,store-rebalancer,obj=‹cpu›] 9  transferring lease to s2

I'll disable the store rebalancer.