search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.06k stars 3.8k forks source link

server: TestStartupInjectedFailureSingleNode failed [session based leasing issues] #118520

Closed cockroach-teamcity closed 8 months ago

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed on master @ 03525787deade5395bac0a8b5735fa7e715e22ef:

                              |   |     src/runtime/asm_amd64.s:1650
                              | Wraps: (2) list-tenants
                              | Wraps: (3) unexpected value: raw_bytes:"\372N\360\267\n3\002" timestamp:<wall_time:1706680561085812090 > 
                              | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
                            Wraps: (4) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2187
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (5) list-tenants
                            Wraps: (6) unexpected value: raw_bytes:"\372N\360\267\n3\002" timestamp:<wall_time:1706680561085812090 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStartupInjectedFailureSingleNode1240926317
--- FAIL: TestStartupInjectedFailureSingleNode (3.54s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/unowned

This test on roachdash | Improve this report!

Jira issue: CRDB-35766

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 719e3abd2172325b89f9d5b82bf34d19c64d9d92:

                              |   |     src/runtime/asm_amd64.s:1650
                              | Wraps: (2) list-tenants
                              | Wraps: (3) unexpected value: raw_bytes:"\213U4)\n3\002" timestamp:<wall_time:1706703417625920013 > 
                              | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
                            Wraps: (4) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2187
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (5) list-tenants
                            Wraps: (6) unexpected value: raw_bytes:"\213U4)\n3\002" timestamp:<wall_time:1706703417625920013 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode2676452708
--- FAIL: TestStartupInjectedFailureSingleNode (34.68s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 07adffbb0c8efef899f39983d66fb87b962e93b0:

                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_arm64.s:1197
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"\245\016\350$\n3\002" timestamp:<wall_time:1706713865247541535 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
--- FAIL: TestStartupInjectedFailureSingleNode (25.99s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ c46b7bb5966c30d24abb34873373ed25dcf6648e:

                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_arm64.s:1197
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"\226\027\373\252\n3\002" timestamp:<wall_time:1706729481264575050 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/f537c08f6697b7d24ac4f4be9defa70f/logTestStartupInjectedFailureSingleNode3058981422
--- FAIL: TestStartupInjectedFailureSingleNode (2.23s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:

                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"\247zdc\n3\002" timestamp:<wall_time:1706764852474099312 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStartupInjectedFailureSingleNode2139810033
--- FAIL: TestStartupInjectedFailureSingleNode (1.47s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:

                              |   |     src/runtime/asm_amd64.s:1650
                              | Wraps: (2) list-tenants
                              | Wraps: (3) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 > 
                              | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
                            Wraps: (4) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2187
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (5) list-tenants
                            Wraps: (6) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode4163893663
--- FAIL: TestStartupInjectedFailureSingleNode (7.92s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:

                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"M\304L\264\n3\002" timestamp:<wall_time:1706801337407398691 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:541: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode1664611304
--- FAIL: TestStartupInjectedFailureSingleNode (12.12s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

rafiss commented 8 months ago

I can reproduce one of these failures on master (fce4d4723519bc4ca6e9ef5da0ae19960c84752c) with:

./dev test pkg/server --stress -f=TestStartupInjectedFailureSingleNode -- --test_env='COCKROACH_RANDOM_SEED=4800477754099231843'

The stack trace is from list-tenants.

            Error Trace:    github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
            Error:          Received unexpected error:
                            failed to start the server controller: list-tenants: unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 > 
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2190
                              | [...repeated from below...]
                            Wraps: (2) failed to start the server controller
                            Wraps: (3) secondary error attachment
                              | list-tenants: unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 > 
                              | (1) attached stack trace
                              |   -- stack trace:
                              |   | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
                              |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              |   | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |   |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
                              |   |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
                              |   |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
                              |   | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |   |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
                              |   |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
                              |   |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |   |     github.com/cockroachdb/cockroach/pkg/server/server.go:2187
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |   |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              |   | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |   |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              |   | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |   |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              |   | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |   |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              |   | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |   |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              |   | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |   |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              |   | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |   |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              |   | testing.tRunner
                              |   |     GOROOT/src/testing/testing.go:1595
                              |   | runtime.goexit
                              |   |     src/runtime/asm_amd64.s:1650
                              | Wraps: (2) list-tenants
                              | Wraps: (3) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 > 
                              | Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
                            Wraps: (4) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
                              | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
                              |     github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2187
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (5) list-tenants
                            Wraps: (6) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server

However, other failures above have a stack trace from migration-job-find-already-completed:

    server_startup_test.go:90: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
            Error:          Received unexpected error:
                            migration-job-find-already-completed: unexpected value: raw_bytes:"\245\016\350$\n3\002" timestamp:<wall_time:1706713865247541535 > 
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_arm64.s:1197
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"\245\016\350$\n3\002" timestamp:<wall_time:1706713865247541535 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
rafiss commented 8 months ago

I tried a bisect with: ./dev test pkg/server --stress --count=20 -f=TestStartupInjectedFailureSingleNode -- --test_env='COCKROACH_RANDOM_SEED=4800477754099231843'

98ab69fa91dd42452f76fdc3e8a96abbde05e6d6 is the first bad commit, which adds session based leasing

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed on master @ fce4d4723519bc4ca6e9ef5da0ae19960c84752c:

                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1632
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"]t\034\215\n3\002" timestamp:<wall_time:1706851281523973892 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStartupInjectedFailureSingleNode1393796343
--- FAIL: TestStartupInjectedFailureSingleNode (2.08s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 0778173b902ce6d418b843389d20509b934f451d:

                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1632
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"=5\025\211\n3\002" timestamp:<wall_time:1706858159231607040 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/63dd67dfe92cccaca0d733d912c3499b/logTestStartupInjectedFailureSingleNode4063179124
--- FAIL: TestStartupInjectedFailureSingleNode (2.58s)
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 0778173b902ce6d418b843389d20509b934f451d:

                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
                              | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
                              | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
                              |     github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
                              | github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
                              | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
                              |     github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
                              | github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
                              |     github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
                              | github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1632
                              | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/server.go:2053
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
                              | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
                              |     github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
                              | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
                              |     github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
                              | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
                              |     github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
                              | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
                              |     github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1595
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1650
                            Wraps: (2) migration-job-find-already-completed
                            Wraps: (3) unexpected value: raw_bytes:"\252\021\256\236\n3\002" timestamp:<wall_time:1706881284260054049 > 
                            Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
            Test:           TestStartupInjectedFailureSingleNode
            Messages:       failed to restart server
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode2943690580
--- FAIL: TestStartupInjectedFailureSingleNode (7.34s)

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!