cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.08k stars 3.8k forks source link

ccl/multiregionccl: TestMultiRegionDataDriven failed #80837

Closed cockroach-teamcity closed 2 years ago

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ c9e0194b19a03d55c6be92572aad3fbafc256334:

    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4287/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:12:
        exec-sql [1 args]
        INSERT INTO db.rbr (k, v, crdb_region) VALUES (1, 1, 'us-east-1'), (2, 2, 'us-central-1'), (3, 3, 'us-west-1')
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4287/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:19:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4287/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:22:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4287/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:25:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    testcluster.go:141: condition failed to evaluate within 45s: unexpectedly found 1 active spans:
             0.000ms      0.000ms    === operation:/cockroach.roachpb.Internal/Batch _unfinished:1 span.kind:server
        goroutine 3358961 [running]:
        runtime/debug.Stack()
            GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x6684758, 0xc0124fd380}, 0x4ec4340, 0x6651100)
            github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x6684758, 0xc0124fd380}, 0xc0124fd380)
            github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers(0xc005999700, {0x6651198, 0xc000136028})
            github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:141 +0x352
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func2()
            github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:356 +0x2b
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc00a460900)
            github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1a
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc00a460900, {0x6651198, 0xc000136020})
            github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:563 +0x2cc
        github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.(*datadrivenTestState).cleanup(0xc007cbe0c0, {0x6651198, 0xc000136020})
            github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/datadriven_test.go:439 +0x58
        github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestMultiRegionDataDriven.func1(0xc00d858000, {0xc00d858000, 0x143})
            github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/datadriven_test.go:393 +0x1cd
        github.com/cockroachdb/datadriven.Walk(0xc0124fd380, {0xc00d858000, 0x5a92c0}, 0xc000574d50)
            github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:413 +0x17e
        github.com/cockroachdb/datadriven.Walk.func1(0xc00683a600)
            github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:426 +0xa5
        testing.tRunner(0xc0124fd380, 0xc01edaa060)
            GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
            GOROOT/src/testing/testing.go:1306 +0x35a
    --- FAIL: TestMultiRegionDataDriven/regional_by_row (101.35s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM) Parameters in this failure: - TAGS=bazel,gss,deadlock

Same failure on other branches

- #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

/cc @cockroachdb/multiregion

This test on roachdash | Improve this report!

Jira issue: CRDB-15491

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 01572daaf94f80f81f843723a8b58d80fe128990:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/e069c319bb13cd862ea8df887bc01cfc/logTestMultiRegionDataDriven665713220
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:394: -- test log scope end --
--- FAIL: TestMultiRegionDataDriven (155.39s)
=== RUN   TestMultiRegionDataDriven/regional_by_table
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1259/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1259/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:4:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-central-1", "eu-central-1";
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1259/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:8:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-east-1";
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1259/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:12:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1259/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:16:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1259/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:19: SELECT * FROM db.rbt WHERE k = 1
        expected:
        served locally: true
        served via follower read: false

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (55.65s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM) Parameters in this failure: - TAGS=bazel,gss,deadlock

Same failure on other branches

- #80902 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1.0] - #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 2c264bd12c15b8811553adc9d90f9230065f3ff3:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:79: test logs captured to: /artifacts/tmp/_tmp/e069c319bb13cd862ea8df887bc01cfc/logTestMultiRegionDataDriven1851469525
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:394: -- test log scope end --
--- FAIL: TestMultiRegionDataDriven (53.75s)
=== RUN   TestMultiRegionDataDriven/regional_by_table
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3979/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3979/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:4:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-central-1", "eu-central-1";
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3979/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:8:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-east-1";
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3979/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:12:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3979/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:16:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3979/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:19: SELECT * FROM db.rbt WHERE k = 1
        expected:
        served locally: true
        served via follower read: false

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (15.07s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM) Parameters in this failure: - TAGS=bazel,gss

Same failure on other branches

- #80902 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1.0] - #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ ba1b0d2cd8557620505d5a330bb682ce82be6c2a:

        created by github.com/klauspost/compress/zstd.newBlockDec
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
        goroutine 10403 [chan receive, 2 minutes]:
        github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc0001706c0)
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94
        created by github.com/klauspost/compress/zstd.newBlockDec
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
        goroutine 10404 [chan receive, 2 minutes]:
        github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000170780)
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94
        created by github.com/klauspost/compress/zstd.newBlockDec
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
        goroutine 10405 [chan receive, 2 minutes]:
        github.com/klauspost/compress/zstd.(*blockDec).startDecoder(0xc000171200)
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:211 +0x94
        created by github.com/klauspost/compress/zstd.newBlockDec
            github.com/klauspost/compress/zstd/external/com_github_klauspost_compress/zstd/blockdec.go:117 +0x187
        goroutine 10406 [select]:
        go.opencensus.io/stats/view.(*worker).start(0xc000cd5600)
            go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:276 +0xb9
        created by go.opencensus.io/stats/view.init.0
            go.opencensus.io/stats/view/external/io_opencensus_go/stats/view/worker.go:34 +0x92
        goroutine 3666800 [running]:
        runtime/debug.Stack()
            GOROOT/src/runtime/debug/stack.go:24 +0x65
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin({0x6708158, 0xc006e50000}, 0x4f23c60, 0x66d4c00)
            github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60 +0x5f
        github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon({0x6708158, 0xc006e50000}, 0xc006e50000)
            github.com/cockroachdb/cockroach/pkg/testutils/soon.go:41 +0x4a
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers(0xc00fef8000, {0x66d4c18, 0xc000080070})
            github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:141 +0x352
        github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func2()
            github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:367 +0x2b
        github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close(0xc0032c63f0)
            github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:107 +0x1a
        github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc0032c63f0, {0x66d4c18, 0xc000080038})
            github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:563 +0x2cc
        github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.(*datadrivenTestState).cleanup(0xc0072ba060, {0x66d4c18, 0xc000080038})
            github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/datadriven_test.go:439 +0x58
        github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test.TestMultiRegionDataDriven.func1(0xc01eaf8000, {0xc01eaf8000, 0x143})
            github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl_test/pkg/ccl/multiregionccl/datadriven_test.go:393 +0x1cd
        github.com/cockroachdb/datadriven.Walk(0xc006e50000, {0xc01eaf8000, 0x5a9300}, 0xc000508d08)
            github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:413 +0x17e
        github.com/cockroachdb/datadriven.Walk.func1(0xc00e75b2c0)
            github.com/cockroachdb/datadriven/external/com_github_cockroachdb_datadriven/datadriven.go:426 +0xa5
        testing.tRunner(0xc006e50000, 0xc00ed7e000)
            GOROOT/src/testing/testing.go:1259 +0x102
        created by testing.(*T).Run
            GOROOT/src/testing/testing.go:1306 +0x35a
    --- FAIL: TestMultiRegionDataDriven/regional_by_row (93.63s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM) Parameters in this failure: - TAGS=bazel,gss,deadlock

Same failure on other branches

- #80902 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1.0] - #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 48e48db89eb3cd6bc38f3631364c516181811280:

        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:35:
        trace-sql [1 args]
        SELECT * FROM db.global WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:41:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:46:
        trace-sql [1 args]
        SELECT * FROM db.global WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:52:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:57:
        trace-sql [1 args]
        SELECT * FROM db.global WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:63:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:122: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3800/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:68: SELECT * FROM db.global WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        served locally: false
    --- FAIL: TestMultiRegionDataDriven/global_tables (20.76s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM) Parameters in this failure: - TAGS=bazel,gss

Same failure on other branches

- #80902 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1.0] - #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

This test on roachdash | Improve this report!

jbowens commented 2 years ago

Another flake on a PR branch:

------- Stdout: -------
=== RUN   TestMultiRegionDataDriven/global_tables
    datadriven_test.go:123: 
        /go/src/github.com/cockroachdb/cockroach/pkg/ccl/multiregionccl/testdata/global_tables:24: SELECT * FROM db.global WHERE k = 1
        expected:
        served locally: true
        served via follower read: false

        found:
        served locally: true
        served via follower read: true
    --- FAIL: TestMultiRegionDataDriven/global_tables (35.07s)

https://teamcity.cockroachdb.com/viewLog.html?buildId=5948632&buildTypeId=Cockroach_UnitTests

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 47d34c9dd0c63dfd462e4380a65470f4cf18e3d6:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:162: test logs captured to: /artifacts/tmp/_tmp/e069c319bb13cd862ea8df887bc01cfc/logTestMultiRegionDataDriven226079405
    test_log_scope.go:80: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:402: -- test log scope end --
--- FAIL: TestMultiRegionDataDriven (324.50s)
=== RUN   TestMultiRegionDataDriven/global_tables
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:4:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-west-1", "us-central-1";
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:8:
        exec-sql [1 args]
        CREATE TABLE db.global(k INT PRIMARY KEY, v INT) LOCALITY GLOBAL;
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:12:
        exec-sql [1 args]
        INSERT INTO db.global VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:16:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:19:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3268/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:24: SELECT * FROM db.global WHERE k = 1
        expected:
        served locally: true
        served via follower read: false

        found:
        served locally: true
        served via follower read: true
    --- FAIL: TestMultiRegionDataDriven/global_tables (55.00s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 years ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 41db784cb97d2749b162020c2c821979094f87b1:

        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:63:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:68:
        trace-sql [1 args]
        SELECT * FROM db.global WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:74:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:81:
        trace-sql [1 args]
        SELECT * FROM db.global WHERE k = 1
        ----
        served locally: false
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:86:
        exec-sql [1 args]
        ALTER DATABASE db ADD REGION "eu-central-1"
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:90:
        wait-for-zone-config-changes [6 args]
        <no input to command>
        ----
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:93:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.global WHERE k = 2
        ----
        LEAD_FOR_GLOBAL_READS
    datadriven_test.go:123: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/843/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-dbg/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:98: SELECT * FROM db.global WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        served locally: true
        served via follower read: false
    --- FAIL: TestMultiRegionDataDriven/global_tables (58.91s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #77908 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot branch-release-22.1]

This test on roachdash | Improve this report!