pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 7781ee008e9bbbfc6f1b76ec001f0072429c4f72:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3134197843
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3134197843
--- FAIL: TestAdminDecommissionedOperations (9.58s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:44867: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-05 13:11:51.183208486 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.02s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

/cc @cockroachdb/unowned

This test on roachdash | Improve this report!

Jira issue: CRDB-41906

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 1a26024075e2c21e076fd7be6716874035811cd8:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2792966324
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2792966324
--- FAIL: TestAdminDecommissionedOperations (8.56s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:41947: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-06 13:14:45.584158694 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 06ff271990ec79047fb4739068e98449ad191e0b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2791578978
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2791578978
--- FAIL: TestAdminDecommissionedOperations (9.19s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:39847: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-07 13:08:36.571696093 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.02s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 06ff271990ec79047fb4739068e98449ad191e0b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations435919806
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations435919806
--- FAIL: TestAdminDecommissionedOperations (8.84s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:34121: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-08 13:10:09.808876348 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 06ff271990ec79047fb4739068e98449ad191e0b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3531386869
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3531386869
--- FAIL: TestAdminDecommissionedOperations (3.30s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 06ff271990ec79047fb4739068e98449ad191e0b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3074977417
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3074977417
--- FAIL: TestAdminDecommissionedOperations (8.99s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:41525: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-09 13:13:54.365313444 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.02s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 74d8d18a9ac3fbda5c1feb6155db8ff464a3531e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations4136451315
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations4136451315
--- FAIL: TestAdminDecommissionedOperations (8.42s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:45251: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-10 13:10:25.872314398 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 74d8d18a9ac3fbda5c1feb6155db8ff464a3531e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3778510571
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3778510571
--- FAIL: TestAdminDecommissionedOperations (2.97s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 86cb9e9ca9a215c65d549adb63defd2a223fffd3:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1669379208
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1669379208
--- FAIL: TestAdminDecommissionedOperations (9.09s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:37121: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-11 16:21:27.528935578 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[msbutler]

just hit this: https://mesolite.cluster.engflow.com/invocations/default/a0ec5cce-4696-48e6-997e-f4414873fc8c

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ d053c41115e52192649792d68ddd7eacba141706:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1939929762
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1939929762
--- FAIL: TestAdminDecommissionedOperations (8.54s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:39813: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-12 16:11:17.461144821 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 6e9795051b217efed8901ad586b8b842bc7c654c:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3338365908
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3338365908
--- FAIL: TestAdminDecommissionedOperations (3.37s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[msbutler]

routing this to kv. I think @nvanbenschoten fixed this on master a few months ago with https://github.com/cockroachdb/cockroach/pull/125969

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 8cd4d4c673c66eed6c6524aefd460c2415bb9b46:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2489867038
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2489867038
--- FAIL: TestAdminDecommissionedOperations (8.49s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:37143: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-13 16:07:34.547425477 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[arulajmani]

The failure mode seems similar to the one that @nvanbenschoten fixed a few months ago. The timing of these test failures line up with https://github.com/cockroachdb/cockroach/pull/130045. I'll open a backport https://github.com/cockroachdb/cockroach/pull/130716.

I also noticed there's a few failures for the Drain case littered in here. Those seem to be different.

[arulajmani]

The Drain failures seem to be happening on other branches as well (e.g. https://github.com/cockroachdb/cockroach/issues/130135)

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 52d7f3db720359153214894c6dd3b4f96bd8be6b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2385134443
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2385134443
--- FAIL: TestAdminDecommissionedOperations (3.37s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:44531: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-14 16:10:40.375954168 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 52d7f3db720359153214894c6dd3b4f96bd8be6b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2530572834
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2530572834
--- FAIL: TestAdminDecommissionedOperations (8.22s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:37755: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-14 16:12:10.452387935 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 52d7f3db720359153214894c6dd3b4f96bd8be6b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2698572336
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2698572336
--- FAIL: TestAdminDecommissionedOperations (8.87s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:46033: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-15 16:07:34.730201353 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 52d7f3db720359153214894c6dd3b4f96bd8be6b:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2980668664
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2980668664
--- FAIL: TestAdminDecommissionedOperations (3.25s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ bb717c49c5e01d566f130b1ed937c8286acea11a:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations928126795
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations928126795
--- FAIL: TestAdminDecommissionedOperations (8.50s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:36849: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-16 16:19:53.492176537 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ e1a7cbe2adde82bdbe67c46237f005158c5ecf82:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2043041157
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2043041157
--- FAIL: TestAdminDecommissionedOperations (9.44s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:45401: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-17 13:21:51.188248019 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 37aa7a26bdd73f1dba9cda431799ce2601ca97cc:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1677067778
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
--- FAIL: TestAdminDecommissionedOperations (8.35s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:41167: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-18 13:31:39.863674097 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 37aa7a26bdd73f1dba9cda431799ce2601ca97cc:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3053222834
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3053222834
--- FAIL: TestAdminDecommissionedOperations (3.26s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 5ae18a722f6efad6b8943d2514aed72a6b2e920a:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3423338438
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3423338438
--- FAIL: TestAdminDecommissionedOperations (4.09s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:35721: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-19 06:18:37.560606421 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.02s)

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 5ae18a722f6efad6b8943d2514aed72a6b2e920a:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1192105192
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1192105192
--- FAIL: TestAdminDecommissionedOperations (8.58s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:41109: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-19 13:24:37.314398908 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 5ae18a722f6efad6b8943d2514aed72a6b2e920a:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2146306233
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2146306233
--- FAIL: TestAdminDecommissionedOperations (3.33s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:40167: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-19 13:31:33.414566012 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ be5cf51b910f7b14b7fa887de4c09b61c3db88b7:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3753589641
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3753589641
--- FAIL: TestAdminDecommissionedOperations (8.43s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:43871: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-20 13:38:41.511671843 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1853174707
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1853174707
--- FAIL: TestAdminDecommissionedOperations (8.86s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:44823: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-21 13:30:47.937901935 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1697548086
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1697548086
--- FAIL: TestAdminDecommissionedOperations (3.12s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2745939871
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations2745939871
--- FAIL: TestAdminDecommissionedOperations (8.85s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:43703: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-22 13:18:41.092452222 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1866028923
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1866028923
--- FAIL: TestAdminDecommissionedOperations (3.39s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:41271: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-22 13:24:42.746957322 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1489885098
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations1489885098
--- FAIL: TestAdminDecommissionedOperations (8.12s)
=== RUN   TestAdminDecommissionedOperations/Events
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0xd
                            actual  : 0x7
            Test:           TestAdminDecommissionedOperations/Events
            Messages:       rpc error: code = PermissionDenied desc = admin-events: RangeIterator failed to seek to /Table/12/2: failed to connect to n1 at 127.0.0.1:44619: initial connection heartbeat failed: grpc: n2 was permanently removed from the cluster at 2024-09-23 13:21:32.925095333 +0000 UTC; it is not allowed to rejoin the cluster [code 7/PermissionDenied], node is decommissioned
    --- FAIL: TestAdminDecommissionedOperations/Events (0.01s)

Parameters:

• TAGS=bazel,gss,deadlock
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[cockroach-teamcity]

pkg/server/storage_api/storage_api_test.TestAdminDecommissionedOperations failed with artifacts on release-23.2 @ 021942d1a7a71bd30fe3aefff6f6a4470cee2a0e:

=== RUN   TestAdminDecommissionedOperations
    test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3380706432
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:93: cluster virtualization disabled due to issue: #81590 (expected label: C-bug)

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:819: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:827: (TestAdminDecommissionedOperations)
    NOTICE: .NodeID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().NodeID() instead.

pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:833: (TestAdminDecommissionedOperations.func1)
    NOTICE: .DB() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().DB() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization
    decommission_test.go:977: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/800529510c8a77f305555fe4848c57f0/logTestAdminDecommissionedOperations3380706432
--- FAIL: TestAdminDecommissionedOperations (3.05s)
=== RUN   TestAdminDecommissionedOperations/Drain
    decommission_test.go:972: 
            Error Trace:    pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:972
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:75
                                        github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:81
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:60
                                        github.com/cockroachdb/cockroach/pkg/testutils/soon.go:42
                                        pkg/server/storage_api/storage_api_test_test/pkg/server/storage_api/decommission_test.go:954
            Error:          Not equal: 
                            expected: 0x7
                            actual  : 0x2
            Test:           TestAdminDecommissionedOperations/Drain
            Messages:       rpc error: code = Unknown desc = some sessions did not respond to cancellation within 1s
    --- FAIL: TestAdminDecommissionedOperations/Drain (1.00s)

Parameters:

• TAGS=bazel,gss
• stress=true
  Help

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130686 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.12-rc release-blocker] - #130514 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-23.2.11-rc release-blocker] - #130388 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130135 pkg/server/storage_api/storage_api_test: TestAdminDecommissionedOperations failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

[nvanbenschoten]

Closed by https://github.com/cockroachdb/cockroach/pull/130716.