search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.17k stars 3.82k forks source link

pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed #131604

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ a0080c77406a238d9c33b450d9a48e749b98ffd1:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1826859878
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1826859878
--- FAIL: TestAdminDecommissionedOperations (3.20s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

/cc @cockroachdb/unowned

This test on roachdash | Improve this report!

Jira issue: CRDB-42628

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 74520af5a4280d723e7434684cca170c6c7b9d5a:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1155654759
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1155654759
--- FAIL: TestAdminDecommissionedOperations (3.25s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 822e2ceb7e98353dbd3d524fed007ac67eaae28d:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2217998935
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2217998935
--- FAIL: TestAdminDecommissionedOperations (3.21s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 713a1609245143c86c1a2e140005eb4042d9f4c3:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1029989510
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1029989510
--- FAIL: TestAdminDecommissionedOperations (3.29s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ f9f40a62700ac2ca5e2777e4cd41d2605c7c421b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations944201711
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations944201711
--- FAIL: TestAdminDecommissionedOperations (3.19s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 23d8c510dfab0da81e2093bfde00dfcc6891df99:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3531348607
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3531348607
--- FAIL: TestAdminDecommissionedOperations (3.36s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 92557346e77333122055c2b7f2b5464cd074ae66:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/e3bcaec89ce770232ceace15c7e08c46/logTestAdminDecommissionedOperations3735932830
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e3bcaec89ce770232ceace15c7e08c46/logTestAdminDecommissionedOperations3735932830
--- FAIL: TestAdminDecommissionedOperations (4.21s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 986119f253a579fd561f03d93e59ec56bdb234d7:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2517827209
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2517827209
--- FAIL: TestAdminDecommissionedOperations (3.31s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 52510b25c4d9261e52ba2d0956d8d103011de62e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3430178526
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3430178526
--- FAIL: TestAdminDecommissionedOperations (3.14s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 9646dcd5e9c8c53b279ebfbf0b147754ba42c9b4:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations4025156112
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations4025156112
--- FAIL: TestAdminDecommissionedOperations (2.94s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 46b9e20d01d0fb84e32b95f79462b6aadd76f34d:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2784559284
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2784559284
--- FAIL: TestAdminDecommissionedOperations (3.39s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 520b62634e9478db36c5ec81d43d1ccaf821185c:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2651055625
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2651055625
--- FAIL: TestAdminDecommissionedOperations (3.48s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 520b62634e9478db36c5ec81d43d1ccaf821185c:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3007107370
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3007107370
--- FAIL: TestAdminDecommissionedOperations (3.29s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 520b62634e9478db36c5ec81d43d1ccaf821185c:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2315071764
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2315071764
--- FAIL: TestAdminDecommissionedOperations (3.25s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ cfd3f4277ded0a42b8ffcb957e543aa2698bd971:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2349469892
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2349469892
--- FAIL: TestAdminDecommissionedOperations (3.19s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 757a6e62c6e6884725d239b5673b4f7a5c830107:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations982468529
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations982468529
--- FAIL: TestAdminDecommissionedOperations (3.40s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 2fe60af0a968ba0e33911716437bfe9f5cfa457e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3336739483
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3336739483
--- FAIL: TestAdminDecommissionedOperations (3.44s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 8cf9b412b06d66b41e8ef1988e062e58b36a807d:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1655806197
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1655806197
--- FAIL: TestAdminDecommissionedOperations (3.16s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 05b4173957118b45d1967234bcb4b399ea348630:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3874138282
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3874138282
--- FAIL: TestAdminDecommissionedOperations (3.28s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 530aac2d73acf3f7b3eab39d98cfd8f1223a5b10:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1663324498
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1663324498
--- FAIL: TestAdminDecommissionedOperations (3.12s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 67813ff8fb532c84782d26af5fab728a3d130f16:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1487625695
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
--- FAIL: TestAdminDecommissionedOperations (3.29s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 3f6131443eb5b80f6a9a8aff70ecc35898192ee0:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3588554782
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3588554782
--- FAIL: TestAdminDecommissionedOperations (3.31s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.01s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ a13ff5f1d2563612f2d15183f8e435ca85a08e2c:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations4191153014
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations4191153014
--- FAIL: TestAdminDecommissionedOperations (3.05s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ a13ff5f1d2563612f2d15183f8e435ca85a08e2c:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1957306267
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1957306267
--- FAIL: TestAdminDecommissionedOperations (3.20s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 0cb9fcf97b09a4caa07393d48a917fd680a9818e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2671800676
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2671800676
--- FAIL: TestAdminDecommissionedOperations (3.28s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ ad4f7954735d179b79c01a748b6f04e5071d56f0:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations310012439
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations310012439
--- FAIL: TestAdminDecommissionedOperations (3.28s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 83445ffc968d8daf9f9dcca8bd9df1bfe5f2eb24:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2872927576
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2872927576
--- FAIL: TestAdminDecommissionedOperations (3.27s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.01s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 days ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 0391542a84ccbe978ac91bed51faa747e1d68dc5:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/e3bcaec89ce770232ceace15c7e08c46/logTestAdminDecommissionedOperations2219051592
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e3bcaec89ce770232ceace15c7e08c46/logTestAdminDecommissionedOperations2219051592
--- FAIL: TestAdminDecommissionedOperations (3.29s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #135598 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.16-rc release-blocker] - #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ a974701fbfca5e39e7067eb3abc1c57efd6510e2:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/e3bcaec89ce770232ceace15c7e08c46/logTestAdminDecommissionedOperations1394152094
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/e3bcaec89ce770232ceace15c7e08c46/logTestAdminDecommissionedOperations1394152094
--- FAIL: TestAdminDecommissionedOperations (4.49s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.01s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #135598 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.16-rc release-blocker] - #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.7-rc release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 73c743b8c97abf9233fb3e612a46a84da1775483:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1338115041
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1338115041
--- FAIL: TestAdminDecommissionedOperations (3.26s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #135598 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot release-blocker] - #134761 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot release-blocker] - #133853 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.15-rc release-blocker] - #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.6-rc release-blocker] - #132275 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131831 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.13-rc release-blocker] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker] - #131296 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker]

This test on roachdash | Improve this report!

kvoli commented 2 days ago

This appears to fail most reliably out of the variants, so I'll use this as the root issue.

kvoli commented 2 days ago

Reproduces fairly quickly with:

dev test pkg/server/storage_api -f TestAdminDecommissionedOperations -v --stress
kvoli commented 2 days ago

The metamorphic vars on a repro for reference:

Details

``` initialized metamorphic constant "max-batch-byte-size" with value 15088426 initialized metamorphic constant "parse-json-impl" with value 0 initialized metamorphic constant "coldata-batch-size" with value 296 initialized metamorphic constant "span-reuse-rate" with value 72 initialized metamorphic constant "COCKROACH_ENABLE_HDR_HISTOGRAMS" with value true initialized metamorphic constant "default-batch-bytes-limit" with value 56718 initialized metamorphic constant "kv.snapshot.ingest_as_write_threshold" with value apply_to_elastic initialized metamorphic constant "use-mux-rangefeed" with value true initialized metamorphic constant "spilling-queue-initial-len" with value 10 initialized metamorphic constant "merge-joiner-groups-buffer" with value 4 initialized metamorphic constant "row-container-rows-per-chunk-shift" with value 1 initialized metamorphic constant "lock-table-iters-before-seek" with value 1 initialized metamorphic constant "mvcc-incremental-iter-tbi" with value false initialized metamorphic constant "storage.value_blocks.enabled" with value false initialized metamorphic constant "storage.ingest_as_flushable.enabled" with value false initialized metamorphic constant "mvcc-max-iters-before-seek" with value 2 initialized metamorphic constant "disable-checksstconflicts-range-key-masking" with value true initialized metamorphic constant "addsst-rewrite-concurrency" with value 14 initialized metamorphic constant "addsst-rewrite-forced" with value true initialized metamorphic constant "split-scans-right-for-stats-first" with value true initialized metamorphic constant "raft-log-truncation-clearrange-threshold" with value 872641 initialized metamorphic constant "kv.snapshot.ingest_as_write_threshold" with value 102400 initialized metamorphic constant "kv_rangefeed_scheduler_enabled" with value true initialized metamorphic constant "kv-batch-size" with value 1 initialized metamorphic constant "datum-row-converter-batch-size" with value 1 initialized metamorphic constant "ColIndexJoin-batch-size" with value 1728691 initialized metamorphic constant "ColIndexJoin-using-streamer-batch-size" with value 4805536 initialized metamorphic constant "inverted-joiner-batch-size" with value 1 initialized metamorphic constant "resolve-membership-single-scan-enabled" with value false initialized metamorphic constant "copy-batch-size" with value 13201 initialized metamorphic constant "async-IE-result-channel-buffer-size" with value 7 initialized metamorphic constant "tail-call-optimization-enabled" with value false initialized metamorphic constant "multiple_active_portals_enabled" with value true ```

Logs for the drain op on a failure:

Details

``` I241121 17:12:42.149056 7211 1@server/drain.go:165 ⋮ [T1,Vsystem,n2] 259 drain request received with doDrain = true, shutdown = false I241121 17:12:42.149079 7211 server/drain.go:346 ⋮ [T1,Vsystem,n2] 260 all tenant servers stopped I241121 17:12:42.149090 7211 1@server/drain.go:392 ⋮ [T1,Vsystem,n2] 261 waiting for health probes to notice that the node is not ready for new sql connections I241121 17:12:42.149099 7211 1@util/log/event_log.go:27 ⋮ [T1,Vsystem,n2] 262 ={"Timestamp":1732209162149098000,"EventType":"node_shutdown_connection_timeout","NodeID":2,"Detail":"draining SQL queries after waiting for server.shutdown.connections.timeout","ConnectionsRemaining":1,"TimeoutMillis":0} I241121 17:12:42.149113 7211 1@sql/pgwire/server.go:651 ⋮ [T1,Vsystem,n2] 263 starting draining SQL connections I241121 17:12:42.149122 7211 1@util/log/event_log.go:27 ⋮ [T1,Vsystem,n2] 264 ={"Timestamp":1732209162149122000,"EventType":"node_shutdown_transaction_timeout","NodeID":2,"Detail":"forcibly closing SQL connections after waiting for server.shutdown.transactions.timeout","ConnectionsRemaining":1,"TimeoutMillis":0} W241121 17:12:42.149139 2787 jobs/registry.go:1127 ⋮ [T1,Vsystem,n2] 265 canceling all adopted jobs due to graceful drain request I241121 17:12:42.149154 2780 sql/stats/automatic_stats.go:568 ⋮ [T1,Vsystem,n2] 266 draining auto stats refresher I241121 17:12:42.149164 2781 sql/stats/automatic_stats.go:620 ⋮ [T1,Vsystem,n2] 267 draining stats garbage collector E241121 17:12:42.149292 7504 2@rpc/peer.go:666 ⋮ [T1,Vsystem,n2,rnode=1,raddr=‹127.0.0.1:64957›,class=system,rpc] 268 failed connection attempt‹ (last connected 145ms ago)›: grpc: ‹n2 was permanently removed from the cluster at 2024-11-21 17:12:41.166718 +0000 UTC; it is not allowed to rejoin the cluster› [code 7/PermissionDenied], node is decommissioned I241121 17:12:42.166870 7599 rpc/heartbeat.go:169 ⋮ [-] 269 failing ping request from node n2 I241121 17:12:42.172597 7548 rpc/heartbeat.go:169 ⋮ [-] 270 failing ping request from node n2 I241121 17:12:42.179590 7558 rpc/heartbeat.go:169 ⋮ [-] 271 failing ping request from node n2 I241121 17:12:42.182805 7661 rpc/heartbeat.go:169 ⋮ [-] 272 failing ping request from node n2 I241121 17:12:42.215046 7703 rpc/heartbeat.go:169 ⋮ [-] 273 failing ping request from node n2 I241121 17:12:42.225745 7723 rpc/heartbeat.go:169 ⋮ [-] 274 failing ping request from node n2 I241121 17:12:42.238412 7750 rpc/heartbeat.go:169 ⋮ [-] 275 failing ping request from node n2 I241121 17:12:42.268898 7735 rpc/heartbeat.go:169 ⋮ [-] 276 failing ping request from node n2 I241121 17:12:42.293027 7796 rpc/heartbeat.go:169 ⋮ [-] 277 failing ping request from node n2 W241121 17:12:42.297202 345 2@gossip/gossip.go:1413 ⋮ [T1,Vsystem,n1] 278 no incoming or outgoing connections I241121 17:12:42.297264 7797 gossip/client.go:108 ⋮ [T1,Vsystem,n1] 279 unvalidated bootstrap gossip dial to ‹127.0.0.1:65057› I241121 17:12:42.312728 7797 gossip/client.go:132 ⋮ [T1,Vsystem,n1] 280 started gossip client to n0 (‹127.0.0.1:65057›) I241121 17:12:42.312832 7848 rpc/heartbeat.go:169 ⋮ [-] 281 failing ping request from node n2 I241121 17:12:42.388149 7886 rpc/heartbeat.go:169 ⋮ [-] 282 failing ping request from node n2 I241121 17:12:42.405635 7898 rpc/heartbeat.go:169 ⋮ [-] 283 failing ping request from node n2 I241121 17:12:42.421786 7917 rpc/heartbeat.go:169 ⋮ [-] 284 failing ping request from node n2 I241121 17:12:42.433904 7956 rpc/heartbeat.go:169 ⋮ [-] 285 failing ping request from node n2 I241121 17:12:42.450074 7950 rpc/heartbeat.go:169 ⋮ [-] 286 failing ping request from node n2 I241121 17:12:42.462058 8009 rpc/heartbeat.go:169 ⋮ [-] 287 failing ping request from node n2 I241121 17:12:42.490535 7997 rpc/heartbeat.go:169 ⋮ [-] 288 failing ping request from node n2 I241121 17:12:42.502714 8040 rpc/heartbeat.go:169 ⋮ [-] 289 failing ping request from node n2 I241121 17:12:42.524330 8065 rpc/heartbeat.go:169 ⋮ [-] 290 failing ping request from node n2 I241121 17:12:42.765175 7553 rpc/heartbeat.go:169 ⋮ [-] 291 failing ping request from node n2 I241121 17:12:42.825829 8127 rpc/heartbeat.go:169 ⋮ [-] 292 failing ping request from node n2 I241121 17:12:42.842248 8173 rpc/heartbeat.go:169 ⋮ [-] 293 failing ping request from node n2 I241121 17:12:42.850780 8174 rpc/heartbeat.go:169 ⋮ [-] 294 failing ping request from node n2 I241121 17:12:42.906825 8196 rpc/heartbeat.go:169 ⋮ [-] 295 failing ping request from node n2 I241121 17:12:42.919005 8253 rpc/heartbeat.go:169 ⋮ [-] 296 failing ping request from node n2 I241121 17:12:42.940337 8201 rpc/heartbeat.go:169 ⋮ [-] 297 failing ping request from node n2 I241121 17:12:42.968538 7976 gossip/client.go:108 ⋮ [T1,Vsystem,n2] 298 unvalidated bootstrap gossip dial to ‹127.0.0.1:64957› W241121 17:12:42.968581 7976 gossip/client.go:126 ⋮ [T1,Vsystem,n2] 299 failed to start gossip client to ‹127.0.0.1:64957›: grpc: ‹n2 was permanently removed from the cluster at 2024-11-21 17:12:41.166718 +0000 UTC; it is not allowed to rejoin the cluster› [code 7/PermissionDenied], node is decommissioned I241121 17:12:42.968645 2392 1@gossip/gossip.go:1427 ⋮ [T1,Vsystem,n2] 300 node has connected to cluster via gossip I241121 17:12:42.968691 2392 kv/kvserver/stores.go:277 ⋮ [T1,Vsystem,n2] 301 wrote 1 node addresses to persistent storage I241121 17:12:42.975960 8306 rpc/heartbeat.go:169 ⋮ [-] 302 failing ping request from node n2 E241121 17:12:42.976272 7977 2@rpc/peer.go:666 ⋮ [T1,Vsystem,n2,rnode=?,raddr=‹127.0.0.1:64957›,class=system,rpc] 303 failed connection attempt‹ (last connected 985ms ago)›: grpc: ‹n2 was permanently removed from the cluster at 2024-11-21 17:12:41.166718 +0000 UTC; it is not allowed to rejoin the cluster› [code 7/PermissionDenied], node is decommissioned I241121 17:12:43.149886 7211 1@server/drain.go:313 ⋮ [T1,Vsystem,n2] 304 drain remaining: 1 I241121 17:12:43.149978 7211 1@server/drain.go:315 ⋮ [T1,Vsystem,n2] 305 drain details: SQL clients: 1 E241121 17:12:43.149995 7211 1@server/drain.go:171 ⋮ [T1,Vsystem,n2] 306 drain failed: some sessions did not respond to cancellation within 1s I241121 17:12:43.150194 3296 1@sql/pgwire/server.go:1367 ⋮ [T1,Vsystem,n2,client=127.0.0.1:65102,hostssl,user=root] 307 closing existing connection while server is draining I241121 17:12:43.150268 3296 4@util/log/event_log.go:27 ⋮ [T1,Vsystem,n2,client=127.0.0.1:65102,hostssl,user=root] 308 ={"Timestamp":1732209163150266000,"EventType":"client_session_end","InstanceID":2,"Network":"tcp","RemoteAddress":"‹127.0.0.1:65102›","SessionID":"180a0b0dee7b43680000000000000002","Duration":2009340625} ```

Logs for the drain op on a passing run:

Details

``` I241121 19:16:56.287153 8065 1@server/drain.go:165 ⋮ [T1,Vsystem,n2] 286 drain request received with doDrain = true, shutdown = false I241121 19:16:56.287192 8065 server/drain.go:346 ⋮ [T1,Vsystem,n2] 287 all tenant servers stopped I241121 19:16:56.287205 8065 1@server/drain.go:392 ⋮ [T1,Vsystem,n2] 288 waiting for health probes to notice that the node is not ready for new sql connections I241121 19:16:56.287215 8065 1@util/log/event_log.go:27 ⋮ [T1,Vsystem,n2] 289 ={"Timestamp":1732216616287213000,"EventType":"node_shutdown_connection_timeout","NodeID":2,"Detail":"draining SQL queries after waiting for server.shutdown.connections.timeout","ConnectionsRemaining":1,"TimeoutMillis":0} I241121 19:16:56.287229 8065 1@sql/pgwire/server.go:651 ⋮ [T1,Vsystem,n2] 290 starting draining SQL connections I241121 19:16:56.287245 8065 1@util/log/event_log.go:27 ⋮ [T1,Vsystem,n2] 292 ={"Timestamp":1732216616287244000,"EventType":"node_shutdown_transaction_timeout","NodeID":2,"Detail":"forcibly closing SQL connections after waiting for server.shutdown.transactions.timeout","ConnectionsRemaining":1,"TimeoutMillis":0} W241121 19:16:56.287233 2843 jobs/registry.go:1127 ⋮ [T1,Vsystem,n2] 291 canceling all adopted jobs due to graceful drain request I241121 19:16:56.287257 2836 sql/stats/automatic_stats.go:568 ⋮ [T1,Vsystem,n2] 293 draining auto stats refresher I241121 19:16:56.287264 2837 sql/stats/automatic_stats.go:620 ⋮ [T1,Vsystem,n2] 294 draining stats garbage collector I241121 19:16:56.319439 8165 rpc/heartbeat.go:169 ⋮ [-] 295 failing ping request from node n2 I241121 19:16:56.332547 8194 rpc/heartbeat.go:169 ⋮ [-] 296 failing ping request from node n2 I241121 19:16:56.669020 8205 rpc/heartbeat.go:169 ⋮ [-] 297 failing ping request from node n2 I241121 19:16:56.679560 8231 rpc/heartbeat.go:169 ⋮ [-] 298 failing ping request from node n2 I241121 19:16:56.733998 8257 rpc/heartbeat.go:169 ⋮ [-] 299 failing ping request from node n2 I241121 19:16:56.743384 8170 rpc/heartbeat.go:169 ⋮ [-] 300 failing ping request from node n2 I241121 19:16:56.793609 8311 rpc/heartbeat.go:169 ⋮ [-] 301 failing ping request from node n2 I241121 19:16:56.821663 8288 gossip/client.go:108 ⋮ [T1,Vsystem,n2] 302 unvalidated bootstrap gossip dial to ‹127.0.0.1:53005› W241121 19:16:56.821773 8288 gossip/client.go:126 ⋮ [T1,Vsystem,n2] 303 failed to start gossip client to ‹127.0.0.1:53005›: grpc: ‹n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster› [code 7/PermissionDenied], node is decommissioned I241121 19:16:56.821935 2393 1@gossip/gossip.go:1427 ⋮ [T1,Vsystem,n2] 304 node has connected to cluster via gossip I241121 19:16:56.822073 2393 kv/kvserver/stores.go:277 ⋮ [T1,Vsystem,n2] 305 wrote 1 node addresses to persistent storage I241121 19:16:56.839754 8295 rpc/heartbeat.go:169 ⋮ [-] 306 failing ping request from node n2 E241121 19:16:56.840396 8289 2@rpc/peer.go:666 ⋮ [T1,Vsystem,n2,rnode=?,raddr=‹127.0.0.1:53005›,class=system,rpc] 307 failed connection attempt‹ (last connected 1.006s ago)›: grpc: ‹n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster› [code 7/PermissionDenied], node is decommissioned I241121 19:16:56.951294 3344 1@sql/pgwire/server.go:1367 ⋮ [T1,Vsystem,n2,client=127.0.0.1:53022,hostssl,user=root] 308 closing existing connection while server is draining I241121 19:16:56.951430 3344 4@util/log/event_log.go:27 ⋮ [T1,Vsystem,n2,client=127.0.0.1:53022,hostssl,user=root] 309 ={"Timestamp":1732216616951419000,"EventType":"client_session_end","InstanceID":2,"Network":"tcp","RemoteAddress":"‹127.0.0.1:53022›","SessionID":"180a11d567eeb3600000000000000002","Duration":2004611708} I241121 19:16:56.951567 8065 sql/sqlstats/persistedsqlstats/flush.go:69 ⋮ [T1,Vsystem,n2] 310 flushing 30 stmt/txn fingerprints (194560 bytes) after 2562047h47m16.854775807s E241121 19:16:56.958123 8065 sql/sqlstats/persistedsqlstats/flush.go:84 ⋮ [T1,Vsystem,n2] 311 encountered an error at flush, checking for statement statistics size limit: ‹fetch-stmt-count›: error in retrieving descs between 1732216612.152224000,0, 1732216614.233157000,0: RangeIterator failed to seek to /Table/3/1/‹1›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.970841 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 312 ‹failed to flush transaction statistics›: flushing transaction ‹10950060388645779556›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:56.971606 8296 rpc/heartbeat.go:169 ⋮ [-] 313 failing ping request from node n2 W241121 19:16:56.972087 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 314 ‹failed to flush statement statistics›: flush statement ‹6348203192952515288›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.972761 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 315 ‹failed to flush transaction statistics›: flushing transaction ‹2199384427631701038›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.973523 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 316 ‹failed to flush statement statistics›: flush statement ‹177636696958847028›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.974631 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 317 ‹failed to flush transaction statistics›: flushing transaction ‹430843496277831770›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.974976 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 318 ‹failed to flush statement statistics›: flush statement ‹12819062270244084721›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.976070 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 319 ‹failed to flush transaction statistics›: flushing transaction ‹3372529360643831805›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.976474 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 320 ‹failed to flush statement statistics›: flush statement ‹14921957443287757346›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.977436 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 321 ‹failed to flush transaction statistics›: flushing transaction ‹15575511678448993431›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:56.977515 8305 rpc/heartbeat.go:169 ⋮ [-] 322 failing ping request from node n2 W241121 19:16:56.978052 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 323 ‹failed to flush statement statistics›: flush statement ‹12292879387043088261›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.978761 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 324 ‹failed to flush transaction statistics›: flushing transaction ‹5567832602717342263›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.979521 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 325 ‹failed to flush statement statistics›: flush statement ‹9344437996277039138›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.980061 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 326 ‹failed to flush transaction statistics›: flushing transaction ‹6160281827883030848›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.980961 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 327 ‹failed to flush statement statistics›: flush statement ‹9405701553376681072›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.981451 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 328 ‹failed to flush transaction statistics›: flushing transaction ‹12321857747860713943›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.982249 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 329 ‹failed to flush statement statistics›: flush statement ‹1879849972839045049›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.982902 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 330 ‹failed to flush transaction statistics›: flushing transaction ‹3485920355470251487›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.983558 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 331 ‹failed to flush statement statistics›: flush statement ‹6719086618408996310›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:56.983889 8366 rpc/heartbeat.go:169 ⋮ [-] 332 failing ping request from node n2 W241121 19:16:56.984292 8343 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 333 ‹failed to flush transaction statistics›: flushing transaction ‹294560657230213835›'s statistics: ‹insert-txn-stats›: RangeIterator failed to seek to /Table/3/1/‹43›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.984870 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 334 ‹failed to flush statement statistics›: flush statement ‹1464507226451452290›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.986390 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 335 ‹failed to flush statement statistics›: flush statement ‹8594249169360073544›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.988007 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 336 ‹failed to flush statement statistics›: flush statement ‹16296089486315486696›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.989446 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 337 ‹failed to flush statement statistics›: flush statement ‹15220787591856491738›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:56.990323 8092 rpc/heartbeat.go:169 ⋮ [-] 338 failing ping request from node n2 W241121 19:16:56.990740 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 339 ‹failed to flush statement statistics›: flush statement ‹18022854731361332895›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.992019 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 340 ‹failed to flush statement statistics›: flush statement ‹316295526215599624›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.993393 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 341 ‹failed to flush statement statistics›: flush statement ‹11458221042986263040›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.994751 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 342 ‹failed to flush statement statistics›: flush statement ‹12354992361753969940›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:56.995925 8489 rpc/heartbeat.go:169 ⋮ [-] 343 failing ping request from node n2 W241121 19:16:56.996322 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 344 ‹failed to flush statement statistics›: flush statement ‹4077394443941908411›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.997714 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 345 ‹failed to flush statement statistics›: flush statement ‹9384234118490468149›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:56.999597 8342 sql/sqlstats/persistedsqlstats/flush.go:193 ⋮ [T1,Vsystem,n2] 346 ‹failed to flush statement statistics›: flush statement ‹15237859555065754932›'s statistics: ‹insert-stmt-stats›: RangeIterator failed to seek to /Table/3/1/‹42›/‹2›/‹1›: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:56.999814 8065 sql/sqlstats/persistedsqlstats/provider.go:133 ⋮ [T1,Vsystem,n2] 347 stopping persisted SQL stats tasks I241121 19:16:56.999837 8065 sql/sqlstats/persistedsqlstats/provider.go:139 ⋮ [T1,Vsystem,n2] 348 persisted SQL stats tasks successfully shut down I241121 19:16:56.999847 8065 sql/stats/automatic_stats.go:398 ⋮ [T1,Vsystem,n2] 349 starting to wait for auto-stats tasks to shut down I241121 19:16:56.999856 8065 sql/stats/automatic_stats.go:401 ⋮ [T1,Vsystem,n2] 350 auto-stats tasks successfully shut down I241121 19:16:56.999864 8065 jobs/registry.go:2090 ⋮ [T1,Vsystem,n2] 351 starting to wait for job registry to shut down I241121 19:16:56.999872 8065 jobs/registry.go:2093 ⋮ [T1,Vsystem,n2] 352 job registry tasks successfully shut down W241121 19:16:57.000389 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 353 error releasing lease ID=12 ver=1 expiration={2024-11-21 19:21:57.108041 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.000840 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 354 error releasing lease ID=20 ver=1 expiration={2024-11-21 19:21:54.902576 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.001200 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 355 error releasing lease ID=11 ver=1 expiration={2024-11-21 19:21:49.637099 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.001531 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 356 error releasing lease ID=8 ver=1 expiration={2024-11-21 19:21:54.224176 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.001856 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 357 error releasing lease ID=1 ver=1 expiration={2024-11-21 19:22:07.185077 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:57.002270 8539 rpc/heartbeat.go:169 ⋮ [-] 358 failing ping request from node n2 W241121 19:16:57.002307 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 359 error releasing lease ID=37 ver=1 expiration={2024-11-21 19:22:00.602777 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.002659 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 360 error releasing lease ID=35 ver=1 expiration={2024-11-21 19:21:47.095112 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.002957 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 361 error releasing lease ID=15 ver=1 expiration={2024-11-21 19:21:54.642748 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.003250 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 362 error releasing lease ID=51 ver=1 expiration={2024-11-21 19:21:42.574397 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.003542 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 363 error releasing lease ID=31 ver=1 expiration={2024-11-21 19:22:01.996573 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.003834 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 364 error releasing lease ID=6 ver=1 expiration={2024-11-21 19:21:44.522337 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.004149 8065 sql/catalog/lease/storage.go:233 ⋮ [T1,Vsystem,n2] 365 error releasing lease ID=40 ver=1 expiration={2024-11-21 19:21:47.424124 +0000 UTC}: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› W241121 19:16:57.004184 2802 sql/sqlliveness/slinstance/slinstance.go:344 ⋮ [T1,Vsystem,n2] 366 exiting heartbeat loop I241121 19:16:57.004200 2802 sql/sqlliveness/slinstance/slinstance.go:329 ⋮ [T1,Vsystem,n2] 367 draining heartbeat loop I241121 19:16:57.004516 8065 1@server/drain.go:313 ⋮ [T1,Vsystem,n2] 368 drain remaining: 13 I241121 19:16:57.004527 8065 1@server/drain.go:315 ⋮ [T1,Vsystem,n2] 369 drain details: SQL clients: 1, descriptor leases: 12 E241121 19:16:57.004534 8065 1@server/drain.go:171 ⋮ [T1,Vsystem,n2] 370 drain failed: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned› I241121 19:16:57.004763 39 testutils/testcluster/testcluster.go:142 ⋮ [-] 371 TestCluster quiescing nodes ```

kvoli commented 2 days ago

I'm not certain this is the relevant diff making the test fail, but it appears most likely that some SQL sessions failing to respond to the cancellation within 1s:

I241121 17:12:43.149886 7211 1@server/drain.go:313 ⋮ [T1,Vsystem,n2] 304  drain remaining: 1
I241121 17:12:43.149978 7211 1@server/drain.go:315 ⋮ [T1,Vsystem,n2] 305  drain details: SQL clients: 1
E241121 17:12:43.149995 7211 1@server/drain.go:171 ⋮ [T1,Vsystem,n2] 306  drain failed: some sessions did not respond to cancellation within 1s

We don't see that log in the passing scenario. Which seems logical, since this is also the error message associated with the CI failures above.

For clarity, The drain is supposed to fail here, its just that its supposed to reach out to another node (I believe) and fail the initial RPC heartbeat on:

E241121 19:16:57.004534 8065 1@server/drain.go:171 ⋮ [T1,Vsystem,n2] 370  drain failed: failed to connect to n1 at ‹127.0.0.1:53005›: ‹initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-11-21 19:16:54.959925 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned›

I'm unsure if this is just relevant to the test, since in either case the drain fails and maybe that is what the original implementers intended? In any case, either the test or non-test code need to change to silence this failure.

Removing the release blocker and marking as a bug.

cockroach-teamcity commented 2 days ago

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 892c4a4823d7882b94d9fdcad04be6146d197d8b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:165: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2245091928
    test_log_scope.go:76: use -show-logs to present logs inline
    test_server_shim.go:88: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:814: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:822: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:828: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:972: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2245091928
--- FAIL: TestAdminDecommissionedOperations (3.21s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:967: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:967
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:70
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:55
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:37
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:949
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #133555 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot P-2 T-kv X-duplicate branch-release-24.1] - #133026 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot P-2 T-kv X-duplicate branch-release-24.1.6-rc] - #131403 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot P-2 T-kv X-duplicate branch-release-24.1]

This test on roachdash | Improve this report!