roachtest: follower-reads/mixed-version/single-region failed

[cockroach-teamcity]

roachtest.follower-reads/mixed-version/single-region failed with artifacts on release-24.3 @ bca9d708828f2e202df23b343efb8119a610c030:

(follower_reads.go:374).runFollowerReadsTest: fewer than 2 follower reads occurred: saw [6 5 0 2] before and [6 129 0 2] after
(mixedversion.go:732).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/follower-reads/mixed-version/single-region/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=2
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-44306

[arulajmani]

Deployment mode:    shared-process
Mutators:           cluster_setting[kv.snapshot_receiver.excise.enabled]
Plan:
├── start cluster at version "v23.1.7" (1)
├── wait for all nodes (:1-4) to acknowledge cluster version '23.1' on system tenant (2)
├── start shared-process tenant "mixed-version-tenant-hqx6b" (3)
├── wait for all nodes (:1-4) to acknowledge cluster version '23.1' on mixed-version-tenant-hqx6b tenant (4)
├── set cluster setting "server.controller.default_tenant" to 'mixed-version-tenant-hqx6b' on system tenant (5)
├── set cluster setting "server.secondary_tenants.authorization.mode" to 'allow-all' on system tenant (6)
├── run "init database" (7)
├── upgrade cluster from "v23.1.7" to "v23.2.6"

@andrewbaptist is it surprising that we're starting this upgrade from 23.1 even though https://github.com/cockroachdb/cockroach/pull/133092 has been merged and backported to release-24.3? I'll assign it to you to investigate.

Either way, removing the release blocker label given this one is run under shared process multi-tenancy and we know that this test is relying on patches that aren't on 23.1.