Closed cockroach-teamcity closed 8 months ago
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 719e3abd2172325b89f9d5b82bf34d19c64d9d92:
| | src/runtime/asm_amd64.s:1650
| Wraps: (2) list-tenants
| Wraps: (3) unexpected value: raw_bytes:"\213U4)\n3\002" timestamp:<wall_time:1706703417625920013 >
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2187
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (5) list-tenants
Wraps: (6) unexpected value: raw_bytes:"\213U4)\n3\002" timestamp:<wall_time:1706703417625920013 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode2676452708
--- FAIL: TestStartupInjectedFailureSingleNode (34.68s)
Parameters:
TAGS=bazel,gss,deadlock
stress=true
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 07adffbb0c8efef899f39983d66fb87b962e93b0:
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_arm64.s:1197
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"\245\016\350$\n3\002" timestamp:<wall_time:1706713865247541535 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
--- FAIL: TestStartupInjectedFailureSingleNode (25.99s)
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ c46b7bb5966c30d24abb34873373ed25dcf6648e:
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_arm64.s:1197
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"\226\027\373\252\n3\002" timestamp:<wall_time:1706729481264575050 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/f537c08f6697b7d24ac4f4be9defa70f/logTestStartupInjectedFailureSingleNode3058981422
--- FAIL: TestStartupInjectedFailureSingleNode (2.23s)
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
server.TestStartupInjectedFailureSingleNode failed on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"\247zdc\n3\002" timestamp:<wall_time:1706764852474099312 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStartupInjectedFailureSingleNode2139810033
--- FAIL: TestStartupInjectedFailureSingleNode (1.47s)
Parameters:
attempt=1
run=3
shard=14
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:
| | src/runtime/asm_amd64.s:1650
| Wraps: (2) list-tenants
| Wraps: (3) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 >
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2187
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (5) list-tenants
Wraps: (6) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode4163893663
--- FAIL: TestStartupInjectedFailureSingleNode (7.92s)
Parameters:
TAGS=bazel,gss,deadlock
stress=true
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ cc4fdffa8532d16544c48ef036689763f737dc6b:
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"M\304L\264\n3\002" timestamp:<wall_time:1706801337407398691 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:541: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode1664611304
--- FAIL: TestStartupInjectedFailureSingleNode (12.12s)
Parameters:
TAGS=bazel,gss
stress=true
I can reproduce one of these failures on master (fce4d4723519bc4ca6e9ef5da0ae19960c84752c) with:
./dev test pkg/server --stress -f=TestStartupInjectedFailureSingleNode -- --test_env='COCKROACH_RANDOM_SEED=4800477754099231843'
The stack trace is from list-tenants
.
Error Trace: github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
Error: Received unexpected error:
failed to start the server controller: list-tenants: unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 >
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2190
| [...repeated from below...]
Wraps: (2) failed to start the server controller
Wraps: (3) secondary error attachment
| list-tenants: unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 >
| (1) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| | github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
| | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| | github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| | github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| | github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| | github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
| | github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
| | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
| | github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
| | github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| | github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| | github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
| | github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
| | github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
| | github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
| | github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| | github.com/cockroachdb/cockroach/pkg/server/server.go:2187
| | github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| | github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| | github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| | github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| | github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| | github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| | github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| | github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| | github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| | github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| | github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| | github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| | testing.tRunner
| | GOROOT/src/testing/testing.go:1595
| | runtime.goexit
| | src/runtime/asm_amd64.s:1650
| Wraps: (2) list-tenants
| Wraps: (3) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 >
| Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Wraps: (4) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).getExpectedRunningTenants
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:206
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers.func1
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:99
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).startInitialSecondaryTenantServers
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:95
| github.com/cockroachdb/cockroach/pkg/server.(*serverController).start
| github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:43
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2187
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (5) list-tenants
Wraps: (6) unexpected value: raw_bytes:"\312\207\345\005\n3\002" timestamp:<wall_time:1706799449780626334 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *secondary.withSecondaryError (4) *withstack.withStack (5) *errutil.withPrefix (6) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
However, other failures above have a stack trace from migration-job-find-already-completed
:
server_startup_test.go:90:
Error Trace: github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
Error: Received unexpected error:
migration-job-find-already-completed: unexpected value: raw_bytes:"\245\016\350$\n3\002" timestamp:<wall_time:1706713865247541535 >
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).execInternal.func1.1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:1074
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1633
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/aarch64-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_arm64.s:1197
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"\245\016\350$\n3\002" timestamp:<wall_time:1706713865247541535 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
I tried a bisect with: ./dev test pkg/server --stress --count=20 -f=TestStartupInjectedFailureSingleNode -- --test_env='COCKROACH_RANDOM_SEED=4800477754099231843'
98ab69fa91dd42452f76fdc3e8a96abbde05e6d6 is the first bad commit, which adds session based leasing
server.TestStartupInjectedFailureSingleNode failed on master @ fce4d4723519bc4ca6e9ef5da0ae19960c84752c:
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1632
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"]t\034\215\n3\002" timestamp:<wall_time:1706851281523973892 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestStartupInjectedFailureSingleNode1393796343
--- FAIL: TestStartupInjectedFailureSingleNode (2.08s)
Parameters:
attempt=1
run=4
shard=14
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 0778173b902ce6d418b843389d20509b934f451d:
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1632
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"=5\025\211\n3\002" timestamp:<wall_time:1706858159231607040 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/63dd67dfe92cccaca0d733d912c3499b/logTestStartupInjectedFailureSingleNode4063179124
--- FAIL: TestStartupInjectedFailureSingleNode (2.58s)
See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)
server.TestStartupInjectedFailureSingleNode failed with artifacts on master @ 0778173b902ce6d418b843389d20509b934f451d:
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next.func1
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:508
| github.com/cockroachdb/cockroach/pkg/sql.(*rowsIterator).Next
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:567
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).queryInternalBuffered
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:681
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowExWithCols
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:733
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRowEx
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:719
| github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).QueryRow
| github.com/cockroachdb/cockroach/pkg/sql/internal.go:703
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable.CheckIfMigrationCompleted
| github.com/cockroachdb/cockroach/pkg/upgrade/migrationstable/migrations_table.go:118
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades.func2
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:220
| github.com/cockroachdb/cockroach/pkg/util/startup.RunIdempotentWithRetryEx[...]
| github.com/cockroachdb/cockroach/pkg/util/startup/retry.go:142
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager.(*Manager).RunPermanentUpgrades
| github.com/cockroachdb/cockroach/pkg/upgrade/upgrademanager/manager.go:216
| github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart
| github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1632
| github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/server.go:2053
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:807
| github.com/cockroachdb/cockroach/pkg/server.(*testServer).Start
| github.com/cockroachdb/cockroach/pkg/server/testserver.go:883
| github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).Start
| github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:15
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect.func1
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1766
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServerWithInspect
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1776
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).RestartServer
| github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:1690
| github.com/cockroachdb/cockroach/pkg/server_test.TestStartupInjectedFailureSingleNode
| github.com/cockroachdb/cockroach/pkg/server_test/pkg/server/server_startup_test.go:90
| testing.tRunner
| GOROOT/src/testing/testing.go:1595
| runtime.goexit
| src/runtime/asm_amd64.s:1650
Wraps: (2) migration-job-find-already-completed
Wraps: (3) unexpected value: raw_bytes:"\252\021\256\236\n3\002" timestamp:<wall_time:1706881284260054049 >
Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *kvpb.ConditionFailedError
Test: TestStartupInjectedFailureSingleNode
Messages: failed to restart server
panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/5b2c9b3a394428c7572d34050aad8975/logTestStartupInjectedFailureSingleNode2943690580
--- FAIL: TestStartupInjectedFailureSingleNode (7.34s)
Parameters:
TAGS=bazel,gss,deadlock
stress=true
server.TestStartupInjectedFailureSingleNode failed on master @ 03525787deade5395bac0a8b5735fa7e715e22ef:
Parameters:
attempt=1
run=10
shard=14
Help
See also: How To Investigate a Go Test Failure (internal)
/cc @cockroachdb/unownedThis test on roachdash | Improve this report!
Jira issue: CRDB-35766