search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.1k stars 3.81k forks source link

server: TestClusterConnectivity failed #130513

Closed cockroach-teamcity closed 1 month ago

cockroach-teamcity commented 1 month ago

server.TestClusterConnectivity failed with artifacts on release-23.2.11-rc @ c70f43425e66787952e3252f84bc6340c128709d:

      github.com/cockroachdb/cockroach/pkg/server/server.go:1998 +0x5024
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:807 +0xe4
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x73
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func2()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:405 +0xad
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func7()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:406 +0x41

Goroutine 162841 (running) created at:
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:404 +0x4fd
  github.com/cockroachdb/cockroach/pkg/server_test.TestClusterConnectivity.func2()
      github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/connectivity_test.go:287 +0x772
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/server_test.TestClusterConnectivity.func2()
      github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/connectivity_test.go:198 +0x526
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44

Goroutine 162840 (running) created at:
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:404 +0x4fd
  github.com/cockroachdb/cockroach/pkg/server_test.TestClusterConnectivity.func2()
      github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/connectivity_test.go:287 +0x772
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.NewTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:364 +0x1444
  github.com/cockroachdb/cockroach/pkg/server_test.TestClusterConnectivity.func2()
      github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/connectivity_test.go:198 +0x526
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
==================

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130486 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-24.2 release-blocker] - #130484 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130415 server: TestClusterConnectivity failed [C-test-failure O-robot T-server-and-security branch-release-23.1 release-blocker] - #130239 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-23.2 release-blocker] - #130238 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-24.1 release-blocker]

/cc @cockroachdb/unowned

This test on roachdash | Improve this report!

Jira issue: CRDB-42100

cockroach-teamcity commented 1 month ago

server.TestClusterConnectivity failed with artifacts on release-23.2.11-rc @ 701e687d9491360b9f080c07bcb70e463184d2e4:

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.
=== RUN   TestClusterConnectivity/topology=2
    test_server_shim.go:146: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)

pkg/server_test/pkg/server/connectivity_test.go:210: (TestClusterConnectivity.func2.1.1)
    NOTICE: .AdvRPCAddr() called via implicit interface ApplicationLayerInterface;
HINT: consider using .ApplicationLayer().AdvRPCAddr() instead.
TIP: consider replacing the test server initialization from:
    ts, ... := serverutils.StartServer(t, ...)
    defer ts.Stopper().Stop(...)
to:
    srv, ... := serverutils.StartServer(t, ...)
    defer srv.Stopper().Stop(...)
    ts := srv.ApplicationLayer()

See also: https://go.crdb.dev/p/testserver-and-cluster-virtualization

pkg/server_test/pkg/server/connectivity_test.go:232: (TestClusterConnectivity.func2.1.2)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

pkg/server_test/pkg/server/connectivity_test.go:248: (TestClusterConnectivity.func2.1.3)
    NOTICE: .StorageClusterID() called via implicit interface StorageLayerInterface;
HINT: consider using .StorageLayer().StorageClusterID() instead.

Parameters:

See also: How To Investigate a Go Test Failure (internal)

Same failure on other branches

- #130486 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-24.2 release-blocker] - #130484 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-24.1.5-rc release-blocker] - #130415 server: TestClusterConnectivity failed [C-test-failure O-robot T-server-and-security branch-release-23.1 release-blocker] - #130239 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-23.2 release-blocker] - #130238 server: TestClusterConnectivity failed [C-test-failure O-robot branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

spilchen commented 1 month ago

I'm closing this one off as 23.2.11 has been published and the rc branch has been removed. The issue only impacts testing, and there’s no issue with the server itself since the license enforcer is currently disabled. The problem only arises due to parallel test servers running in the same process.