search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.03k stars 3.8k forks source link

ccl/multiregionccl: TestMultiRegionDataDriven failed #98020

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ fcea283ebca17a6d923c5d4b0401697438b77dbd:

=== RUN   TestMultiRegionDataDriven
I230304 07:16:59.107940 1 (gostd) rand.go:199  [T1] 1  random seed: 4267267524760381632
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/a5ee0b0d8f7b79a0b4e56a682326700f/logTestMultiRegionDataDriven2123913141
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/a5ee0b0d8f7b79a0b4e56a682326700f/logTestMultiRegionDataDriven2123913141
--- FAIL: TestMultiRegionDataDriven (203.33s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9326/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:

        expected:

        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (131.32s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

/cc @cockroachdb/sql-schema

This test on roachdash | Improve this report!

Jira issue: CRDB-25019

ajwerner commented 1 year ago

This is a manifestation of https://github.com/cockroachdb/cockroach/issues/92235.

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4dc9e98d8cc1b137ee4cc950a4a42e5895bb890b:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2274/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (42.21s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4dc9e98d8cc1b137ee4cc950a4a42e5895bb890b:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1706893346
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1706893346
--- FAIL: TestMultiRegionDataDriven (884.32s)
=== RUN   TestMultiRegionDataDriven/regional_by_table
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1462/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1462/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:4:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-central-1", "eu-central-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1462/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:8:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-east-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1462/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:12:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1462/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:16:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1462/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:19:
         SELECT * FROM db.rbt WHERE k = 1
        expected:
        served locally: true
        served via follower read: false

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (200.18s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 01e14f68f7c757cab1c878cebfecd73c5119046b:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/2955/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (42.16s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

msbutler commented 1 year ago

this flaked on me https://teamcity.cockroachdb.com/viewLog.html?buildId=9007206&buildTypeId=Cockroach_BazelEssentialCi

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 5b2a5670cbbe895d76602c230390816e783e0caa:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3262/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (43.32s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 95684388bfbb372ac8c7e1a8ebd4a0d6f447e147:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/8168/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (41.59s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 95684388bfbb372ac8c7e1a8ebd4a0d6f447e147:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/30/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (43.02s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ e4924e2b9be4a36d466beab53a80df9241df4783:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3728/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (42.76s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

ajwerner commented 1 year ago

This failure more seems new as of four days ago.

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 36d39aadc5e7b88732e1318363177507193eb52e:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3678/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (46.47s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4215b86090794a78649aa4463c7bf95d74c57c37:

    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
        trace-sql [1 args]
        SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3729/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:232:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (43.29s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

rafiss commented 1 year ago

I used ./dev test --stress pkg/ccl/multiregionccl/ -f=TestMultiRegionDataDriven/regional_by_table --timeout=3m to reproduce this and bisect.

It pointed to 3a4e0c8f543259a7fb1a819687bb431406e6dc4b as the first failing commit.

The failure is:

            expected:
            served locally: true
            served via follower read: true

            found:
            recording contains > 1 dist sender send messages

cc @matthewtodd in case you have any ideas. I wonder if the changes in pkg/sql/executor_statement_metrics.go are relevant to this test.

rafiss commented 1 year ago

and cc @maryliag since 3a4e0c8 was done as part of @cockroachdb/cluster-observability work.

matthewtodd commented 1 year ago

Thanks, @rafiss, I'm guessing there's something in this resolver.GetAllInstances call that may occasionally use the dist sender. Perhaps the test assertion just needs to be loosened up a bit?

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 6c99966f604f3521acdb925b9f689529ffd46df3:

        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
        trace-sql [1 args]
        SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:232:
        trace-sql [1 args]
        SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        ----
        served locally: true
        served via follower read: true
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/11673/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:238:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (77.08s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 53dbb86acb1d48309530181b94838faf937084d3:

        ALTER DATABASE db DROP REGION "us-west-1"
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:187:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:194:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:200:
        wait-for-zone-config-changes [5 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:204:
        sleep-for-follower-read [0 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:209:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:214:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:219:
        refresh-range-descriptor-cache [2 args]
        SELECT * FROM db.rbt WHERE k = 1
        ----
        LAG_BY_CLUSTER_SETTING
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3359/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:226:
         SELECT * FROM db.rbt AS OF SYSTEM TIME follower_read_timestamp() WHERE k = 1
        expected:
        served locally: true
        served via follower read: true

        found:
        recording contains > 1 dist sender send messages
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (43.93s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 9acc75317aebfdfe69fb097a8a28e0040c7a67fd:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1394509355
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1394509355
--- FAIL: TestMultiRegionDataDriven (195.43s)
=== RUN   TestMultiRegionDataDriven/regional_by_table
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/617/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/617/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:4:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-central-1", "eu-central-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/617/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:8:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-east-1";
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/617/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:12:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:129: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/617/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_table:16:

        expected:

        found:
        expected leaseholder 0 but got 1
    --- FAIL: TestMultiRegionDataDriven/regional_by_table (145.95s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

knz commented 1 year ago

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9149058?showRootCauses=false&expandBuildChangesSection=true&expandBuildProblemsSection=true&expandBuildTestsSection=true

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 1e9899da8cb250a4e560b280beb2b0805ee75a78:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/92210d9178e507c2b00f19c1ad4bbd36/logTestMultiRegionDataDriven1937418003
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:431: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/92210d9178e507c2b00f19c1ad4bbd36/logTestMultiRegionDataDriven1937418003
--- FAIL: TestMultiRegionDataDriven (205.55s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven_test.go:132: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/9156/execroot/com_github_cockroachdb_cockroach/bazel-out/aarch64-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:

        expected:

        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (144.60s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

yuzefovich commented 1 year ago

I saw a related failure on staging (after #99484):

=== RUN   TestMultiRegionDataDriven/regional_by_row
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:4:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-west-1", "us-central-1";
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:8:
        exec-sql [1 args]
        CREATE TABLE db.rbr(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY ROW;
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:12:
        exec-sql [1 args]
        INSERT INTO db.rbr (k, v, crdb_region) VALUES (1, 1, 'us-east-1'), (2, 2, 'us-central-1'), (3, 3, 'us-west-1')
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:19:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:22:
        wait-for-zone-config-changes [8 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4186/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:25:

        expected:

        found:
        could not get leaseholder: no valid lease
    --- FAIL: TestMultiRegionDataDriven/regional_by_row (167.01s)

It looks different though, right?

matthewtodd commented 1 year ago

Thanks, @yuzefovich, yeah, that looks like a different cause than the regional_by_table and secondary_region failures I've been looking at. 🤔 (And that un-skipping PR wouldn't have introduced the regional_by_row failure you saw.) I'll keep an eye on it and skip it if it flakes again.

rafiss commented 1 year ago

This flaked for me on a 23.1 backport PR. Do we need to backport some skipping PRs too?

https://teamcity.cockroachdb.com/buildConfiguration/Cockroach_BazelEssentialCi/9326855?showRootCauses=false&expandBuildProblemsSection=true&expandBuildTestsSection=true

matthewtodd commented 1 year ago

It looks like regional_by_row is flaking now, too. I've just sent #100139 to skip it, will backport.

matthewtodd commented 1 year ago

This issue captures 3 tests that are flaky / skipped:

I've been looking today at just the regional_by_table one today, since Rafi's git bisect found my 3a4e0c8f543259a7fb1a819687bb431406e6dc4b as the culprit for its initial "recording contains > 1 dist sender send messages" failure.

I can confirm I see the same failure locally; though I am having trouble making sense of the trace. I can understand which parts of it are triggering the test to fail, but it is not clear to me what they mean or where they are coming from. (I don't see any signs of that Resolver.getAllInstances call I had initially suspected.) Unfortunately I didn't save the trace for review here, but I will capture the next one I see.

(Incidentally, when I run that same test later in the commit history at 5839fe0fd13aa33f247e6591875e197fa451e770, just before it was skipped, I get an "expected leaseholder 0 but got 2" failure on an earlier line. So it appears there may be more than one source of flakiness we'll need to address in this test.)

I am now re-running a git bisect on a GCE worker to see if I can reproduce Rafi's results:

git bisect start \
  5839fe0fd13aa33f247e6591875e197fa451e770 \
  b272795bca772b311aa1082c357267a8486028c6

git bisect run ./dev test --stress \
  pkg/ccl/multiregionccl \
  -f=TestMultiRegionDataDriven/regional_by_table \
  --timeout=3m

Update: This git bisect run confirms Rafi's finding that my 3a4e0c8f543259a7fb1a819687bb431406e6dc4b is the first bad commit.

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ a4ab6b6243f509e62993b4d81855f23470d9901a:

I230419 06:45:12.572887 51905626 kv/kvserver/replica_rangefeed.go:737  [n4,s4,r10/3:/Table/{6-7}] 2  RangeFeed failed to nudge: retry rangefeed (REASON_NO_LEASEHOLDER)
I230419 06:45:13.169540 685627 kv/kvserver/store_raft.go:656  [T1,n2,s2,r29/5:/Table/2{7-8},raft] 3  raft ready handling: 0.78s [append=0.00s, apply=0.00s, , other=0.78s], wrote []; node might be overloaded
I230419 06:45:13.173784 770649 kv/kvserver/store_raft.go:656  [T1,n3,s3,r54/5:/Table/5{3-4},raft] 4  raft ready handling: 0.57s [append=0.00s, apply=0.00s, , other=0.57s], wrote []; node might be overloaded
I230419 06:45:13.177904 882869 kv/kvserver/store_raft.go:656  [T1,n4,s4,r59/4:/Table/5{8-9},raft] 5  raft ready handling: 1.60s [append=0.00s, apply=0.00s, non-blocking-sync=0.00s, other=1.60s], wrote [append-batch=258 B, append-ent=195 B (1), ]; node might be overloaded
I230419 06:45:13.250178 1117563 kv/kvserver/store_raft.go:656  [T1,n6,s6,r62/2:/{Table/61-Max},raft] 6  raft ready handling: 0.97s [append=0.00s, apply=0.25s, , other=0.73s], wrote [apply=215 B (1)]; node might be overloaded
I230419 06:45:13.182824 685466 kv/kvserver/store_raft.go:656  [T1,n2,s2,r15/6:/Table/1{3-4},raft] 7  raft ready handling: 0.62s [append=0.00s, apply=0.00s, , other=0.62s], wrote []; node might be overloaded
W230419 06:45:13.179753 2848503 2@rpc/clock_offset.go:276  [T1,n2,rnode=4,raddr=127.0.0.1:34617,class=system,rpc] 10  latency jump (prev avg 126.99ms, current 823.65ms)
I230419 06:45:13.183112 880859 kv/kvserver/store_raft.go:656  [T1,n4,s4,r31/3:/{Table/29-NamespaceTab…},raft] 8  raft ready handling: 0.76s [append=0.00s, apply=0.15s, sync=0.00s, other=0.61s], wrote [append-batch=50 B, apply=213 B (1)]; node might be overloaded
I230419 06:45:13.185363 685211 kv/kvserver/store_raft.go:656  [T1,n2,s2,r7/2:/Table/{3-4},raft] 9  raft ready handling: 0.61s [append=0.00s, apply=0.00s, , other=0.61s], wrote []; node might be overloaded
I230419 06:45:13.212894 770654 kv/kvserver/store_raft.go:656  [T1,n3,s3,r60/5:/Table/{59-60},raft] 12  raft ready handling: 0.60s [append=0.00s, apply=0.00s, , other=0.60s], wrote []; node might be overloaded
I230419 06:45:13.197061 685461 kv/kvserver/store_raft.go:656  [T1,n2,s2,r27/6:/Table/2{5-6},raft] 11  raft ready handling: 0.62s [append=0.00s, apply=0.00s, , other=0.62s], wrote []; node might be overloaded
I230419 06:45:13.193617 881260 kv/kvserver/store_raft.go:656  [T1,n4,s4,r44/4:/Table/4{2-3},raft] 13  raft ready handling: 0.58s [append=0.00s, apply=0.00s, , other=0.58s], wrote []; node might be overloaded
I230419 06:45:13.203929 56613 kv/kvserver/store_raft.go:656  [T1,n1,s1,r55/1:/Table/5{4-5},raft] 14  raft ready handling: 0.89s [append=0.00s, apply=0.00s, , other=0.89s], wrote []; node might be overloaded
I230419 06:45:13.236388 685759 kv/kvserver/store_raft.go:656  [T1,n2,s2,r41/5:/Table/{39-40},raft] 15  raft ready handling: 0.66s [append=0.00s, apply=0.00s, , other=0.66s], wrote []; node might be overloaded
I230419 06:45:13.244100 685111 kv/kvserver/store_raft.go:656  [T1,n2,s2,r54/6:/Table/5{3-4},raft] 16  raft ready handling: 0.71s [append=0.00s, apply=0.00s, , other=0.71s], wrote []; node might be overloaded
W230419 06:45:13.239099 2957738 2@rpc/clock_offset.go:276  [T1,n3,rnode=5,raddr=127.0.0.1:44351,class=default,rpc] 18  latency jump (prev avg 141.26ms, current 729.36ms)
I230419 06:45:13.228411 1033983 kv/kvserver/store_raft.go:656  [T1,n5,s5,r61/3:/Table/6{0-1},raft] 17  raft ready handling: 0.81s [append=0.00s, apply=0.00s, sync=0.00s, other=0.81s], wrote [append-batch=316 B, append-ent=215 B (1), ]; node might be overloaded
I230419 06:45:13.247528 882762 kv/kvserver/store_raft.go:656  [T1,n4,s4,r17/3:/Table/1{5-6},raft] 19  raft ready handling: 0.74s [append=0.00s, apply=0.65s, sync=0.00s, other=0.09s], wrote [append-batch=50 B, apply=219 B (1)]; node might be overloaded
I230419 06:45:13.275536 880928 kv/kvserver/store_raft.go:656  [T1,n4,s4,r1/2:/{Min-System/NodeL…},raft] 20  raft ready handling: 0.86s [append=0.00s, apply=0.00s, sync=0.00s, other=0.85s], wrote [append-batch=614 B, append-ent=462 B (2), ]; node might be overloaded
I230419 06:45:13.283981 770675 kv/kvserver/store_raft.go:656  [T1,n3,s3,r48/5:/Table/4{6-7},raft] 22  raft ready handling: 0.65s [append=0.00s, apply=0.00s, , other=0.65s], wrote []; node might be overloaded
I230419 06:45:13.261624 770872 kv/kvserver/store_raft.go:656  [T1,n3,s3,r50/5:/Table/{48-50},raft] 21  raft ready handling: 0.63s [append=0.00s, apply=0.00s, , other=0.63s], wrote []; node might be overloaded
W230419 06:45:13.278913 2973591 2@rpc/clock_offset.go:276  [T1,n5,rnode=4,raddr=127.0.0.1:34617,class=default,rpc] 23  latency jump (prev avg 185.87ms, current 725.16ms)
I230419 06:45:13.291785 56888 kv/kvserver/store_raft.go:656  [T1,n1,s1,r39/1:/Table/3{7-8},raft] 25  raft ready handling: 0.70s [append=0.00s, apply=0.00s, , other=0.70s], wrote []; node might be overloaded
W230419 06:45:13.297360 3052359 2@rpc/clock_offset.go:276  [T1,n6,rnode=3,raddr=127.0.0.1:33145,class=system,rpc] 24  latency jump (prev avg 157.98ms, current 810.64ms)
I230419 06:45:13.302677 685394 kv/kvserver/store_raft.go:656  [T1,n2,s2,r50/6:/Table/{48-50},raft] 26  raft ready handling: 0.73s [append=0.00s, apply=0.00s, non-blocking-sync=0.00s, other=0.73s], wrote [append-batch=278 B, append-ent=215 B (1), ]; node might be overloaded
I230419 06:45:13.303800 685358 kv/kvserver/store_raft.go:656  [T1,n2,s2,r62/6:/{Table/61-Max},raft] 27  raft ready handling: 0.78s [append=0.00s, apply=0.00s, , other=0.78s], wrote []; node might be overloaded
I230419 06:45:13.303265 685430 kv/kvserver/store_raft.go:656  [T1,n2,s2,r3/5:/System/{NodeLive…-tsd},raft] 28  raft ready handling: 0.90s [append=0.00s, apply=0.80s, sync=0.00s, other=0.10s], wrote [append-batch=50 B, apply=201 KiB (2)]; node might be overloaded
I230419 06:45:13.321916 685632 kv/kvserver/store_raft.go:656  [T1,n2,s2,r35/2:/Table/3{3-4},raft] 29  raft ready handling: 0.69s [append=0.00s, apply=0.00s, , other=0.69s], wrote []; node might be overloaded
I230419 06:45:13.328207 770685 kv/kvserver/store_raft.go:656  [T1,n3,s3,r18/5:/Table/1{6-7},raft] 31  raft ready handling: 0.99s [append=0.00s, apply=0.00s, sync=0.00s, other=0.99s], wrote [append-batch=280 B, append-ent=217 B (1), ]; node might be overloaded
I230419 06:45:13.328031 56506 kv/kvserver/store_raft.go:656  [T1,n1,s1,r6/1:/Table/{0-3},raft] 30  raft ready handling: 1.21s [append=0.00s, apply=1.00s, , other=0.21s], wrote [append-batch=50 B, apply=209 B (1)]; node might be overloaded
I230419 06:45:13.328704 685548 kv/kvserver/store_raft.go:656  [T1,n2,s2,r22/2:/Table/2{0-1},raft] 32  raft ready handling: 0.99s [append=0.00s, apply=0.78s, sync=0.00s, other=0.21s], wrote [append-batch=50 B, apply=215 B (1)]; node might be overloaded
I230419 06:45:13.332898 57271 kv/kvserver/replica_rangefeed.go:711  [T1,n1,s1,r10/1:/Table/{6-7}] 33  RangeFeed closed timestamp 1681886686.429943709,0 is behind by 26.902877702s
I230419 06:45:13.330055 1033327 kv/kvserver/store_raft.go:656  [T1,n5,s5,r60/4:/Table/{59-60},raft] 34  raft ready handling: 2.01s [append=0.00s, apply=0.88s, non-blocking-sync=0.00s, other=1.13s], wrote [append-batch=50 B, apply=195 B (1)]; node might be overloaded
I230419 06:45:13.335784 56862 kv/kvserver/store_raft.go:656  [T1,n1,s1,r3/1:/System/{NodeLive…-tsd},raft] 36  raft ready handling: 1.42s [append=0.00s, apply=1.09s, non-blocking-sync=0.00s, other=0.33s], wrote [append-batch=174 B, append-ent=76 B (1), apply=201 KiB (2)]; node might be overloaded
I230419 06:45:13.338350 1034585 kv/kvserver/store_raft.go:656  [T1,n5,s5,r28/4:/Table/2{6-7},raft] 35  raft ready handling: 1.88s [append=0.00s, apply=0.00s, non-blocking-sync=0.00s, other=1.88s], wrote [append-batch=258 B, append-ent=195 B (1), ]; node might be overloaded
I230419 06:45:13.339922 1033717 kv/kvserver/store_raft.go:656  [T1,n5,s5,r43/3:/Table/4{1-2},raft] 38  raft ready handling: 1.97s [append=0.00s, apply=0.87s, non-blocking-sync=0.00s, other=1.10s], wrote [append-batch=12 B, apply=229 B (1)]; node might be overloaded
I230419 06:45:13.338707 57051 kv/kvserver/store_raft.go:656  [T1,n1,s1,r16/1:/Table/1{4-5},raft] 37  raft ready handling: 0.74s [append=0.00s, apply=0.00s, , other=0.74s], wrote []; node might be overloaded
I230419 06:45:13.333039 685106 kv/kvserver/store_raft.go:656  [T1,n2,s2,r36/5:/Table/3{4-5},raft] 39  raft ready handling: 0.91s [append=0.00s, apply=0.82s, non-blocking-sync=0.00s, other=0.09s], wrote [append-batch=50 B, apply=219 B (1)]; node might be overloaded
I230419 06:45:13.343135 880247 kv/kvserver/store_raft.go:656  [T1,n4,s4,r24/3:/Table/2{2-3},raft] 40  raft ready handling: 0.77s [append=0.00s, apply=0.00s, sync=0.00s, other=0.77s], wrote [append-batch=260 B, append-ent=197 B (1), ]; node might be overloaded
W230419 06:45:13.349707 775353 2@rpc/clock_offset.go:276  [T1,n3,rnode=1,raddr=127.0.0.1:33603,class=default,rpc] 42  latency jump (prev avg 176.78ms, current 810.89ms)
I230419 06:45:13.350215 1034549 kv/kvserver/store_raft.go:656  [T1,n5,s5,r10/4:/Table/{6-7},raft] 41  raft ready handling: 1.15s [append=0.00s, apply=0.98s, sync=0.00s, other=0.17s], wrote [append-batch=50 B, apply=209 B (1)]; node might be overloaded
I230419 06:45:13.363698 1034532 kv/kvserver/store_raft.go:656  [T1,n5,s5,r22/3:/Table/2{0-1},raft] 43  raft ready handling: 1.28s [append=0.00s, apply=1.09s, sync=0.00s, other=0.18s], wrote [append-batch=12 B, apply=215 B (1)]; node might be overloaded
I230419 06:45:13.356465 880456 kv/kvserver/store_raft.go:656  [T1,n4,s4,r30/4:/Table/2{8-9},raft] 44  raft ready handling: 2.30s [append=0.00s, apply=0.00s, non-blocking-sync=0.00s, other=2.30s], wrote [append-batch=262 B, append-ent=199 B (1), ]; node might be overloaded
W230419 06:45:13.366096 882987 kv/kvserver/replica_gossip.go:157  [T1,n4,s4,r2/2:/System/NodeLiveness{-Max}] 46  could not acquire lease for range gossip: replica unavailable: (n4,s4):2 unable to serve request to r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n4,s4):2, (n6,s6):3, (n5,s5):4, next=5, gen=6]: closed timestamp: 1681885713.352503723,0 (2023-04-19 06:28:33); raft status: {"id":"2","term":77,"vote":"1","commit":425,"lead":"0","raftState":"StatePreCandidate","applied":425,"progress":{},"leadtransferee":"0"}: operation "probe" timed out after 1m0.196s (given timeout 1m0s): result is ambiguous: after 60.15s of attempting command: context deadline exceeded
I230419 06:45:13.389495 1117462 kv/kvserver/store_raft.go:656  [T1,n6,s6,r14/4:/Table/1{2-3},raft] 45  raft ready handling: 1.93s [append=0.00s, apply=0.00s, non-blocking-sync=0.00s, other=1.93s], wrote [append-batch=50 B, ]; node might be overloaded
I230419 06:45:13.368767 882399 kv/kvserver/store_raft.go:656  [T1,n4,s4,r34/3:/Table/3{2-3},raft] 47  raft ready handling: 0.88s [append=0.00s, apply=0.87s, , other=0.01s], wrote [apply=219 B (1)]; node might be overloaded
--- FAIL: TestMultiRegionDataDriven (1447.84s)
=== RUN   TestMultiRegionDataDriven/global_tables
    testcluster.go:1469: condition failed to evaluate within 45s: rpc error: code = Internal desc = An internal server error has occurred. Please check your CockroachDB logs for more details. Error Code: 40003
    --- FAIL: TestMultiRegionDataDriven/global_tables (1446.73s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #101520 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1.0] - #99563 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1] - #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 8124bff03d073f35f4d2b6a2048c7f4417d757d9:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven736898380
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven736898380
--- FAIL: TestMultiRegionDataDriven (1189.67s)
=== RUN   TestMultiRegionDataDriven/global_tables
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4552/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven_test.go:124: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/4552/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/global_tables:4:
         CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS  "us-west-1", "us-central-1";
        expected:

        found:
        pq: rpc error: code = Internal desc = An internal server error has occurred. Please check your CockroachDB logs for more details.
    --- FAIL: TestMultiRegionDataDriven/global_tables (1189.66s)

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #101520 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1.0] - #99563 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1] - #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 10665f9856d9c7ca51557bdf5b48fe9df796228b:

github.com/sasha-s/go-deadlock/external/com_github_sasha_s_go_deadlock/deadlock.go:137 go-deadlock.(*RWMutex).RLock ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go:109 kvserver.(*Store).tryGetReplica ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go:172 kvserver.(*Store).tryGetOrCreateReplica ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go:78 kvserver.(*Store).getOrCreateReplica ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:337 kvserver.(*Store).withReplicaForRequest ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:589 kvserver.(*Store).processRequestQueue ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:373 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 824786 lock 0xc00ef360d8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1191 kvserver.(*Replica).tick ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1190 kvserver.(*Replica).tick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:674 kvserver.(*Store).processTick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:386 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 1060019 lock 0xc0051c6d58
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1191 kvserver.(*Replica).tick ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1190 kvserver.(*Replica).tick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:674 kvserver.(*Store).processTick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:386 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 1060383 lock 0xc01597bf00
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1193 kvserver.(*Replica).tick ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1192 kvserver.(*Replica).tick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:674 kvserver.(*Store).processTick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:386 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 720050 lock 0xc0172d11c0
github.com/cockroachdb/cockroach/pkg/util/schedulerlatency/sampler.go:189 schedulerlatency.(*sampler).sampleOnTickAndInvokeCallbacks ??? <<<<<
github.com/cockroachdb/cockroach/pkg/util/schedulerlatency/sampler.go:188 schedulerlatency.(*sampler).sampleOnTickAndInvokeCallbacks ???
github.com/cockroachdb/cockroach/pkg/util/schedulerlatency/sampler.go:143 schedulerlatency.StartSampler.func1 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 963244 lock 0xc00cd5c600
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1193 kvserver.(*Replica).tick ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1192 kvserver.(*Replica).tick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:674 kvserver.(*Store).processTick ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:386 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #101520 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1.0] - #99563 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1] - #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

matthewtodd commented 1 year ago

I've just sent #102192 to address these flakes.

matthewtodd commented 1 year ago

I've sent #102192 to address the flakiness I caused here and to unskip this secondary_region test.

Two other tests, regional_by_row and regional_by_table are still failing under stress, see notes in my comment here.

@rafiss could you handle further triage / investigation from here?

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 5608a146973cb002756eef7bcafd002eb038afb4:

github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:394 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 21284760 lock 0xc000812658
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:712 kvserver.(*Replica).handleRaftReady ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:711 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:645 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:394 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 21284206 lock 0xc00f628658
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:712 kvserver.(*Replica).handleRaftReady ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:711 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:645 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:394 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 21825754 lock 0xc024be89a0
github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:985 mon.(*BytesMonitor).releaseBytes ??? <<<<<
github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:984 mon.(*BytesMonitor).releaseBytes ???
github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:829 mon.(*BoundAccount).Close ???
github.com/cockroachdb/cockroach/pkg/util/mon/bytes_usage.go:812 mon.(*BoundAccount).Clear ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:77 kvserver.(*raftReceiveQueue).drainLocked ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:68 kvserver.(*raftReceiveQueue).Drain ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:579 kvserver.(*Store).processRequestQueue ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:373 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 20408338 lock 0xc0082059d8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:712 kvserver.(*Replica).handleRaftReady ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:711 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:645 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:394 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

goroutine 21669812 lock 0xc00458d9d8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:712 kvserver.(*Replica).handleRaftReady ??? <<<<<
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:711 kvserver.(*Replica).handleRaftReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:645 kvserver.(*Store).processReady ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:394 kvserver.(*raftSchedulerShard).worker ???
github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:299 kvserver.(*raftScheduler).Start.func2 ???
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470 stop.(*Stopper).RunAsyncTaskEx.func2 ???

Parameters: TAGS=bazel,gss,deadlock

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #101520 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1.0] - #99563 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-schema branch-release-23.1] - #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-sessions branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 87c9598a9d59433775764c5a78604c2d3a488ad1:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven867874979
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven867874979
--- FAIL: TestMultiRegionDataDriven (163.01s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3586/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:

        expected:

        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (133.08s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #101520 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-foundations branch-release-23.1.0] - #99563 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-foundations branch-release-23.1] - #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-foundations branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ a65bca232dfde1862e5aff343e7b1f3a379d6cd6:

=== RUN   TestMultiRegionDataDriven
    test_log_scope.go:161: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1148872100
    test_log_scope.go:79: use -show-logs to present logs inline
=== CONT  TestMultiRegionDataDriven
    datadriven_test.go:428: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven1148872100
--- FAIL: TestMultiRegionDataDriven (158.60s)
=== RUN   TestMultiRegionDataDriven/secondary_region
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:1:
        new-cluster [1 args]
        <no input to command>
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:4:
        exec-sql [1 args]
        SET CLUSTER SETTING server.time_until_store_dead = '25s';
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:8:
        exec-sql [1 args]
        CREATE DATABASE db PRIMARY REGION "us-west-1" REGIONS "us-central-1", "eu-west-1" SECONDARY REGION "us-east-1";
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:12:
        exec-sql [1 args]
        ALTER DATABASE db SURVIVE REGION FAILURE;
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:16:
        exec-sql [1 args]
        CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:20:
        exec-sql [1 args]
        INSERT INTO db.rbt VALUES (1, 1), (2,2)
        ----
    datadriven.go:143: 
        /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3814/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/secondary_region:25:

        expected:

        found:
        expected leaseholder 2 but got 3
    --- FAIL: TestMultiRegionDataDriven/secondary_region (130.71s)

Parameters: TAGS=bazel,gss

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #101520 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-foundations branch-release-23.1.0] - #99563 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-foundations branch-release-23.1] - #92235 ccl/multiregionccl: TestMultiRegionDataDriven failed [C-test-failure O-robot T-sql-foundations branch-release-22.2 skipped-test]

This test on roachdash | Improve this report!

matthewtodd commented 1 year ago

Hi, @maryliag and @rafiss -- I think we were in a tricky spot with multiple causes for these flakes. Having rolled back mine, I'm not sure that there's more I can do, so I'm going to unassign myself from this ticket, though I'm happy to pair through more of the bisecting if you like, @rafiss.

matthewtodd commented 1 year ago

Hmm, actually, it's the secondary_region test I un-skipped (after it passed under stress) that's flaking now. I've sent https://github.com/cockroachdb/cockroach/pull/103292 to skip it again.

chengxiong-ruan commented 1 year ago

I tried to repro the errors on master by stressing them on gceworker but with no luck. Is anyone still able to repro them, for example: ./dev test pkg/ccl/multiregionccl --filter 'TestMultiRegionDataDriven/regional_by_table' --stress --stress-args="-p=16"

chengxiong-ruan commented 1 year ago

So we turned on regional_by_row and regional_by_table. secondary_region is still skipped and looks like it's easier to repro now. Specifically, all my repros failed at this same line where the expected replica lease holder is on a different node than expected. I've tried explicitly move the leaseholder by doing something like this:

exec-sql idx=2
create function get_store_id() returns int language sql as $$ select store_id FROM crdb_internal.kv_store_status where node_id = crdb_internal.node_id(); $$
----

exec-sql idx=2
alter range relocate lease to get_store_id() for select range_id from [show ranges from table db.rbt];
----

But that didn't help. @rafiss I doubt that this is something that sql foundation is the team should handle. But I'm not sure if there is a multi-region team, or maybe this is something kv can help.

chengxiong-ruan commented 1 year ago

Looks like the test was actually trying to transfer the lease to the expected node. However, the node is a non-voter for some reason. A lease cannot be transferred to a non-voter. Will investigate more.

chengxiong-ruan commented 1 year ago

@kvoli FWIW, here is a diff I used to print more logs and also zone config.

diff --git a/pkg/ccl/multiregionccl/datadriven_test.go b/pkg/ccl/multiregionccl/datadriven_test.go
index 2f8e49ebd5..081249a58a 100644
--- a/pkg/ccl/multiregionccl/datadriven_test.go
+++ b/pkg/ccl/multiregionccl/datadriven_test.go
@@ -35,6 +35,7 @@ import (
        "github.com/cockroachdb/cockroach/pkg/testutils/serverutils"
        "github.com/cockroachdb/cockroach/pkg/testutils/skip"
        "github.com/cockroachdb/cockroach/pkg/testutils/testcluster"
+       "github.com/cockroachdb/cockroach/pkg/testutils/sqlutils"
        "github.com/cockroachdb/cockroach/pkg/util/leaktest"
        "github.com/cockroachdb/cockroach/pkg/util/log"
        "github.com/cockroachdb/cockroach/pkg/util/syncutil"
@@ -219,6 +220,29 @@ SET CLUSTER SETTING kv.closed_timestamp.propagation_slack = '0.5s'
                                if err != nil {
                                        return err.Error()
                                }
+                       case "query-sql":
+                               mustHaveArgOrFatal(t, d, serverIdx)
+                               var err error
+                               var idx int
+                               d.ScanArgs(t, serverIdx, &idx)
+                               sqlDB, err := ds.getSQLConn(idx)
+                               if err != nil {
+                                       return err.Error()
+                               }
+
+                               rows, err := sqlDB.Query(d.Input)
+                               if err != nil {
+                                       return err.Error()
+                               }
+                               rowsStr, err := sqlutils.RowsToStrMatrix(rows)
+                               if err != nil {
+                                       return err.Error()
+                               }
+                               var strRows []string
+                               for _, row := range rowsStr {
+                                       strRows = append(strRows, strings.Join(row, "\t"))
+                               }
+                               return errors.AssertionFailedf(strings.Join(strRows, "\n")).Error()

                        case "trace-sql":
                                mustHaveArgOrFatal(t, d, serverIdx)
@@ -362,6 +386,7 @@ SET CLUSTER SETTING kv.closed_timestamp.propagation_slack = '0.5s'
                                                        // is a voter. Otherwise, TransferRangeLease will silently fail.
                                                        newLeaseholderType := actualPlacement.getReplicaType(expectedLeaseIdx)
                                                        if newLeaseholderType != replicaTypeVoter {
+                                                               log.Infof(ctx, "expected node %d to be a voter but was %s", expectedLeaseIdx, newLeaseholderType.String())
                                                                return errors.CombineErrors(
                                                                        leaseErr,
                                                                        errors.Newf(
@@ -370,11 +395,13 @@ SET CLUSTER SETTING kv.closed_timestamp.propagation_slack = '0.5s'
                                                                                newLeaseholderType.String()))
                                                        }

-                                                       log.VEventf(
+                                                       log.Infof(
                                                                ctx,
-                                                               2,
                                                                "transferring lease from node %d to %d", actualLeaseIdx, expectedLeaseIdx)
                                                        err = ds.tc.TransferRangeLease(desc, ds.tc.Target(expectedLeaseIdx))
+                                                       if err != nil {
+                                                               log.Infof(ctx, "failed to transfer lease from %d to %d", actualLeaseIdx, expectedLeaseIdx)
+                                                       }
                                                        return errors.CombineErrors(
                                                                leaseErr,
                                                                err,
diff --git a/pkg/ccl/multiregionccl/testdata/secondary_region b/pkg/ccl/multiregionccl/testdata/secondary_region
index 16a3c9fc94..e04ff9d2dd 100644
--- a/pkg/ccl/multiregionccl/testdata/secondary_region
+++ b/pkg/ccl/multiregionccl/testdata/secondary_region
@@ -1,6 +1,3 @@
-skip issue-num=98020
-----
-
 new-cluster localities=us-east-1,us-east-1,us-west-1,us-west-1,us-central-1,us-central-1,us-central-1,eu-west-1,eu-west-1,eu-west-1
 ----

@@ -16,6 +13,10 @@ exec-sql idx=2
 ALTER DATABASE db SURVIVE REGION FAILURE;
 ----

+query-sql idx=2
+select jsonb_pretty(crdb_internal.pb_to_json('cockroach.config.zonepb.ZoneConfig', config)) from system.zones where id = 104;
+----
+
 exec-sql idx=2
 CREATE TABLE db.rbt(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY TABLE IN "us-west-1";
kvoli commented 1 year ago

I reproduced the problem where a non-voter is stuck on n3, when it requires a voter for the lease.

Unsure on the fix yet. The problem is coming from not promoting n3 from non-voter to voter during rebalance. I have a repro here https://github.com/cockroachdb/cockroach/pull/106548/files

kvoli commented 1 year ago

Filed https://github.com/cockroachdb/cockroach/issues/106559 for the underlying cause.

kvoli commented 1 year ago

Adding the A-kv-test-failure-complex label, rationale is https://github.com/cockroachdb/cockroach/issues/106559#issuecomment-1631016264.

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ f295bd861a3a427652b19c2254d2401ebb4a3c8e:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
    panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack: 

goroutine 13353321 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3b7e12a?, {0x7985520, 0xc01344e180})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5c4da80, 0xc014fceb80})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xb53aab8?)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc05e7f8000)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x7985520, 0xc062d56630})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x7985520, 0xc062d56630}, 0x0, 0x1, 0x0?, 0x0?, {0x62a3f94, 0x19}, {0x0, 0x0, ...})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 +0x11f
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc00ea694a0, {0x7985520, 0xc01b8717d0}, {0x79a5298, 0xc043e4e4b0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 +0x12c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x58a0d80?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc00ea68c80, 0x5bc8860?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 +0x4e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 +0x317
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc006166a80, 0x2?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0046b6690, {0x7985520, 0xc01344e180}, {0x7968c60, 0xc006166a80}, 0xc005607500)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x7985520?, 0xc01344e180?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3768/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:8: exec-sql [1 args] CREATE TABLE db.rbr(k INT PRIMARY KEY, v INT) LOCALITY REGIONAL BY ROW; ---- * * ERROR: a panic has occurred! * use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * (1) attached stack trace * -- stack trace: * | runtime.gopanic * | GOROOT/src/runtime/panic.go:884 * | [...repeated from below...] * Wraps: (2) assertion failure * Wraps: (3) attached stack trace * -- stack trace: * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450 * | github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 * | github.com/cockroachdb/cockroach/pkg/util/log.vEventf * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 * | github.com/cockroachdb/cockroach/pkg/util/log.VEvent * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 * | runtime.goexit * | GOROOT/src/runtime/asm_amd64.s:1594 * Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError * ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ f295bd861a3a427652b19c2254d2401ebb4a3c8e:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
    panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack: 

goroutine 15431521 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3b7e12a?, {0x7985520, 0xc017119dd0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5c4da80, 0xc02284e220})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xb53aab8?)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc03e27e800)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x7985520, 0xc00c336030})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x7985520, 0xc00c336030}, 0x0, 0x1, 0xd?, 0x0?, {0x62a3f94, 0x19}, {0x0, 0x0, ...})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 +0x11f
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc00629eda0, {0x7985520, 0xc011c31aa0}, {0x79a5298, 0xc01c0fdea0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 +0x12c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x58a0d80?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc00629e580, 0x5bc8860?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 +0x4e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 +0x317
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc011521500, 0x2?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc016e76d90, {0x7985520, 0xc017119dd0}, {0x7968c60, 0xc011521500}, 0xc0115a8000)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x7985520?, 0xc017119dd0?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` ---- served locally: true served via follower read: true === RUN TestMultiRegionDataDriven/regional_by_row * * ERROR: a panic has occurred! * use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * (1) attached stack trace * -- stack trace: * | runtime.gopanic * | GOROOT/src/runtime/panic.go:884 * | [...repeated from below...] * Wraps: (2) assertion failure * Wraps: (3) attached stack trace * -- stack trace: * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450 * | github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 * | github.com/cockroachdb/cockroach/pkg/util/log.vEventf * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 * | github.com/cockroachdb/cockroach/pkg/util/log.VEvent * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 * | runtime.goexit * | GOROOT/src/runtime/asm_amd64.s:1594 * Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError * ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ e57e9742146b7fac7f7530fa6224e11121ada92f:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
    panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack: 

goroutine 15481771 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3b7e9ca?, {0x7985ee0, 0xc019e3ac90})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5c4e240, 0xc0414b3b50})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0x7985ee0?)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer(...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:225
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).createTracingSpans(0xc01f5cd0d8, {0x7985ee0, 0xc03ac7c810})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:154 +0x4f0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).DecodeAndBind(0xc01f5ccc80?, {0x7985ee0, 0xc03ac7c810}, {0xc043550000?, 0xc?, 0x10?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:63 +0x66
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode(...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:142
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:878 +0x73a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc002901500, 0x2?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0141247e0, {0x7985ee0, 0xc019e3ac90}, {0x7969620, 0xc002901500}, 0xc004576a80)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x7985ee0?, 0xc019e3ac90?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` === RUN TestMultiRegionDataDriven test_log_scope.go:167: test logs captured to: /artifacts/tmp/_tmp/e558fc8050776f4c54ea39ba371b49da/logTestMultiRegionDataDriven2649951294 test_log_scope.go:81: use -show-logs to present logs inline === RUN TestMultiRegionDataDriven/regional_by_row * * ERROR: a panic has occurred! * use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * (1) attached stack trace * -- stack trace: * | runtime.gopanic * | GOROOT/src/runtime/panic.go:884 * | [...repeated from below...] * Wraps: (2) assertion failure * Wraps: (3) attached stack trace * -- stack trace: * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).Tracer * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:225 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).createTracingSpans * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:154 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaDecoder).DecodeAndBind * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_decoder.go:63 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).Decode * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:142 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:878 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 * | runtime.goexit * | GOROOT/src/runtime/asm_amd64.s:1594 * Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError * ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ fcfa15d9694c0b840ac0c5c1b1489fa90a62efd8:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
    panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack: 

goroutine 16044818 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3b7f56a?, {0x7988bc0, 0xc0209024e0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5c50960, 0xc02cb605e0})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xb540ab8?)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc00fdac400)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x7988bc0, 0xc02937dfb0})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x7988bc0, 0xc02937dfb0}, 0x0, 0x1, 0xd?, 0x0?, {0x62a71c3, 0x19}, {0x0, 0x0, ...})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 +0x11f
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc024c414a0, {0x7988bc0, 0xc036256120}, {0x79a8938, 0xc01714dc20})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 +0x12c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x58a3ae0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc024c40c80, 0x5bcb740?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 +0x4e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 +0x317
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0220ff500, 0x2?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00d27dea0, {0x7988bc0, 0xc0209024e0}, {0x796c240, 0xc0220ff500}, 0xc00ba09500)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x7988bc0?, 0xc0209024e0?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` ---- served locally: true served via follower read: true === RUN TestMultiRegionDataDriven/regional_by_row * * ERROR: a panic has occurred! * use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * (1) attached stack trace * -- stack trace: * | runtime.gopanic * | GOROOT/src/runtime/panic.go:884 * | [...repeated from below...] * Wraps: (2) assertion failure * Wraps: (3) attached stack trace * -- stack trace: * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450 * | github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 * | github.com/cockroachdb/cockroach/pkg/util/log.vEventf * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 * | github.com/cockroachdb/cockroach/pkg/util/log.VEvent * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 * | runtime.goexit * | GOROOT/src/runtime/asm_amd64.s:1594 * Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError * ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 8d2d2bf47cb3da1c27baa140a5afb5d28a5ed8f6:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
    panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack: 

goroutine 15580019 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3b81c6a?, {0x7993de0, 0xc00dfc5f20})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5c55b80, 0xc06a1892a0})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xb54eab8?)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc0083d6400)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x7993de0, 0xc009eb5f20})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x7993de0, 0xc009eb5f20}, 0x0, 0x1, 0x0?, 0x0?, {0x62ac7d5, 0x19}, {0x0, 0x0, ...})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 +0x11f
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc019cfc120, {0x7993de0, 0xc01832c5d0}, {0x79b3b58, 0xc019d6b950})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 +0x12c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x58a8a40?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc019cfb900, 0x5bd0800?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 +0x4e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 +0x317
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc00685aa80, 0x2?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc02839f180, {0x7993de0, 0xc00dfc5f20}, {0x7977440, 0xc00685aa80}, 0xc003dc5500)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x7993de0?, 0xc00dfc5f20?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/1173/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:4: exec-sql [1 args] CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS "us-west-1", "us-central-1"; ---- * * ERROR: a panic has occurred! * use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * (1) attached stack trace * -- stack trace: * | runtime.gopanic * | GOROOT/src/runtime/panic.go:884 * | [...repeated from below...] * Wraps: (2) assertion failure * Wraps: (3) attached stack trace * -- stack trace: * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450 * | github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 * | github.com/cockroachdb/cockroach/pkg/util/log.vEventf * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 * | github.com/cockroachdb/cockroach/pkg/util/log.VEvent * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 * | runtime.goexit * | GOROOT/src/runtime/asm_amd64.s:1594 * Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError * ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

ccl/multiregionccl.TestMultiRegionDataDriven failed with artifacts on master @ 4fe2a80d81c6fc5a3da3c7c44c5fc38da67e0367:

Fatal error:

panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish> [recovered]
    panic: use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: <stack not captured. Set debugUseAfterFinish>

Stack: 

goroutine 14776304 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x3b82f4a?, {0x799ac20, 0xc02059f560})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x5c5cbe0, 0xc0209efc40})
    GOROOT/src/runtime/panic.go:884 +0x212
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish(0xb552b80?)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 +0x12e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType(0xc004db7800)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 +0x1e
github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose(...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450
github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog({0x799ac20, 0xc04fab9440})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 +0x3a
github.com/cockroachdb/cockroach/pkg/util/log.vEventf({0x799ac20, 0xc04fab9440}, 0x0, 0x1, 0xd?, 0x0?, {0x62b3a9c, 0x19}, {0x0, 0x0, ...})
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 +0x11f
github.com/cockroachdb/cockroach/pkg/util/log.VEvent(...)
    github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup(0xc02bd78120, {0x799ac20, 0xc0448758c0}, {0x79ba998, 0xc03689eaa0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 +0x12c8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2(0x58af620?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked(0xc02bd77900, 0x5bd77c0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 +0x4e
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 +0x317
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc02c2e0a80, 0x2?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0239722a0, {0x799ac20, 0xc02059f560}, {0x797e280, 0xc02c2e0a80}, 0xc02c2e0000)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 +0x274
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x799ac20?, 0xc02059f560?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` /home/roach/.cache/bazel/_bazel_roach/c5a4e7d36696d9cd970af2045211a7df/sandbox/processwrapper-sandbox/3419/execroot/com_github_cockroachdb_cockroach/bazel-out/k8-fastbuild/bin/pkg/ccl/multiregionccl/multiregionccl_test_/multiregionccl_test.runfiles/com_github_cockroachdb_cockroach/pkg/ccl/multiregionccl/testdata/regional_by_row:4: exec-sql [1 args] CREATE DATABASE db PRIMARY REGION "us-east-1" REGIONS "us-west-1", "us-central-1"; ---- * * ERROR: a panic has occurred! * use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * (1) attached stack trace * -- stack trace: * | runtime.gopanic * | GOROOT/src/runtime/panic.go:884 * | [...repeated from below...] * Wraps: (2) assertion failure * Wraps: (3) attached stack trace * -- stack trace: * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).detectUseAfterFinish * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:182 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).RecordingType * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:442 * | github.com/cockroachdb/cockroach/pkg/util/tracing.(*Span).IsVerbose * | github.com/cockroachdb/cockroach/pkg/util/tracing/span.go:450 * | github.com/cockroachdb/cockroach/pkg/util/log.getSpanOrEventLog * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:92 * | github.com/cockroachdb/cockroach/pkg/util/log.vEventf * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:212 * | github.com/cockroachdb/cockroach/pkg/util/log.VEvent * | github.com/cockroachdb/cockroach/pkg/util/log/trace.go:231 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:601 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:775 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:728 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:416 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 * | runtime.goexit * | GOROOT/src/runtime/asm_amd64.s:1594 * Wraps: (4) use of Span after Finish. Span: /cockroach.roachpb.Internal/Batch. Finish previously called at: * Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError * ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

kvoli commented 1 year ago

Split out issue for the span after use failure https://github.com/cockroachdb/cockroach/issues/108110

EDIT: nvm, there's already an issue https://github.com/cockroachdb/cockroach/issues/107521

kvoli commented 1 year ago

Unassigning myself, as I'm not currently working on this.