ccl/kvccl/kvfollowerreadsccl: TestBoundedStalenessDataDriven failed

[cockroach-teamcity]

ccl/kvccl/kvfollowerreadsccl.TestBoundedStalenessDataDriven failed with artifacts on master @ c11c13ba9efddd2d1453baa405795ca0bb4f9dc5:

    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:136:
        reset-matching-stmt-for-tracing [0 args]
        <no input to command>
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:141:
        exec [0 args]
        SET CLUSTER SETTING kv.closed_timestamp.target_duration = '1hr';
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:145:
        exec [0 args]
        ALTER TABLE t ADD COLUMN new_col INT NOT NULL DEFAULT 2
        ----
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:151:
        query [1 args]
        SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('10s') WHERE pk = 1
        ----
        1 2
        events (1 found):
         * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:158:
        query [1 args]
        SELECT * FROM t AS OF SYSTEM TIME with_min_timestamp(now() - '10s') WHERE pk = 1
        ----
        1 2
        events (1 found):
         * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:165:
        query [1 args]
        SELECT * FROM t AS OF SYSTEM TIME with_max_staleness('10s', false) WHERE pk = 1
        ----
        1 2
        events (1 found):
         * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    datadriven.go:144: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/504/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/kvccl/kvfollowerreadsccl/kvfollowerreadsccl_test_/kvfollowerreadsccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/kvccl/kvfollowerreadsccl/testdata/boundedstaleness/single_row:172:
        query [1 args]
        SELECT * FROM t AS OF SYSTEM TIME with_min_timestamp(now() - '10s', false) WHERE pk = 1
        ----
        1 2
        events (1 found):
         * event 1: colbatchscan trace on node_idx 2: local read then remote leaseholder read
    boundedstaleness_test.go:383: condition failed to evaluate within 45s: from boundedstaleness_test.go:408: not follower reads found:
        events (0 found):
    --- FAIL: TestBoundedStalenessDataDriven/single_row (62.51s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-39841

[arulajmani]

We've seen this test flake rarely; keeping the issue open for now, but tagging it as a P3.

[github-actions[bot]]

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.