search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.17k stars 3.82k forks source link

pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test: TestTenantUpgradeInterlock failed #132711

Open cockroach-teamcity opened 1 month ago

cockroach-teamcity commented 1 month ago

pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test.TestTenantUpgradeInterlock failed with artifacts on release-23.1 @ 25d41095d53d905d6e908cc98696df1326b6c377:

    tenant_upgrade_test.go:317: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) verifying basic SQL functionality
    tenant_upgrade_test.go:322: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) verifying the version of the storage cluster
    tenant_upgrade_test.go:329: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) upgrading the storage cluster
    tenant_upgrade_test.go:332: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) checking the tenant after the storage cluster upgrade
    tenant_upgrade_test.go:336: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) start upgrading the tenant
    tenant_upgrade_test.go:391: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) upgrader is ready
    tenant_upgrade_test.go:393: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) starting another tenant server
    tenant_upgrade_test.go:440: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) waiting for the instance table to get in the right state
    tenant_upgrade_test.go:451: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) resuming upgrade
    tenant_upgrade_test.go:453: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) waiting for upgrade to complete
    tenant_upgrade_test.go:383: unexpected error:  pq: upgrade failed due to transient SQL servers
    tenant_upgrade_test.go:455: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) upgrade completed
    tenant_upgrade_test.go:474: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) check the second server still works
    tenant_upgrade_test.go:476: (TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC) waiting for second server to reach target final version
    sql_runner.go:115: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:128
                                        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:349
                                        pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test_test/pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/tenant_upgrade_test.go:477
                                        pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test_test/pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/tenant_upgrade_test.go:488
            Error:          Received unexpected error:
                            query 'SHOW CLUSTER SETTING version': expected:
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:352
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:66
                              | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                              |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:208
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:72
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:128
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:349
                              | pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test_test.TestTenantUpgradeInterlock.func1
                              |     pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test_test/pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/tenant_upgrade_test.go:477
                              | pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test_test.TestTenantUpgradeInterlock.func2.1
                              |     pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/upgradeinterlockccl_test_test/pkg/ccl/kvccl/kvtenantccl/upgradeinterlockccl/tenant_upgrade_test.go:488
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1446
                              | runtime.goexit
                              |     GOROOT/src/runtime/asm_arm64.s:1172
                            Wraps: (2) query 'SHOW CLUSTER SETTING version': expected:
                              | 23.1
                              |
                              | got:
                              | 22.2
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
        --- FAIL: TestTenantUpgradeInterlock/current_binary_version/pause_after_fence_RPC (78.18s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/multi-tenant

This test on roachdash | Improve this report!

Jira issue: CRDB-43237

rimadeodhar commented 2 weeks ago

Dropping the release-blocker tag since the error is similar to what we have seen before in and seems more of a test environment issue https://github.com/cockroachdb/cockroach/issues/119239.