ccl/multiregionccl: TestMultiRegionDataDriven failed

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd:

=== RUN   TestMultiRegionDataDriven
I230304 07:16:59.107940 1 (gostd) rand.go:199  [T1] 1  random seed: 4267267524760381632
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/a5ee0b0d8f7b79a0b4e56a682326700f/logTestMultiRegionDataDriven2123913141
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/a5ee0b0d8f7b79a0b4e56a682326700f/logTestMultiRegionDataDriven2123913141
--- FAIL: TestMultiRegionDataDriven (203.33s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:

        expected:

        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (131.32s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-25019

cockroachdb / cockroach

ccl/multiregionccl: TestMultiRegionDataDriven failed #98020