sql: TestShowRangesMultipleStores failed

cockroach-teamcity commented 1 month ago

sql.TestShowRangesMultipleStores failed on master @ 1b0a374fd2a101cebcfb24cff4b3b57795ad1df6:

=== RUN   TestShowRangesMultipleStores
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestShowRangesMultipleStores761670925
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    show_ranges_test.go:147: -- test log scope end --
test logs left over in: outputs.zip/logTestShowRangesMultipleStores761670925
--- FAIL: TestShowRangesMultipleStores (123.11s)
=== RUN   TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS
    sql_runner.go:121: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                                        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                                        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
            Error:          Received unexpected error:
                            query '
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:368
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                              | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                              |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:218
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:82
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                              | github.com/cockroachdb/cockroach/pkg/sql_test.TestShowRangesMultipleStores.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (2) query '
                              | SELECT DISTINCT
                              |         (
                              |         array_position(replica_localities, lease_holder_locality)
                              |         = array_position(replicas, lease_holder)
                              |         )
                              |     FROM [SHOW RANGES FROM DATABASE system WITH DETAILS]': expected:
                              | true
                              |
                              | got:
                              | true
                              | NULL
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    --- FAIL: TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS (60.75s)

Parameters:

attempt=1
race=true
run=1
shard=12
Help

_{This test on roachdash | Improve this report!

Jira issue: CRDB-42088}

cockroach-teamcity commented 1 month ago

sql.TestShowRangesMultipleStores failed on master @ 31794d8d89e3e117e1654cb13ad109da04de5445:

=== RUN   TestShowRangesMultipleStores
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestShowRangesMultipleStores733170128
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    show_ranges_test.go:147: -- test log scope end --
test logs left over in: outputs.zip/logTestShowRangesMultipleStores733170128
--- FAIL: TestShowRangesMultipleStores (134.07s)
=== RUN   TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS
    sql_runner.go:121: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                                        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                                        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
            Error:          Received unexpected error:
                            query '
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:368
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                              | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                              |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:218
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:82
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                              | github.com/cockroachdb/cockroach/pkg/sql_test.TestShowRangesMultipleStores.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (2) query '
                              | SELECT DISTINCT
                              |         (
                              |         array_position(replica_localities, lease_holder_locality)
                              |         = array_position(replicas, lease_holder)
                              |         )
                              |     FROM [SHOW RANGES FROM DATABASE system WITH DETAILS]': expected:
                              | true
                              |
                              | got:
                              | true
                              | NULL
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    --- FAIL: TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS (61.02s)

Parameters:

attempt=1
race=true
run=2
shard=12
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

sql.TestShowRangesMultipleStores failed on master @ 3f8226eb7f09eb93c26dda4cd47e023a8f56ea23:

=== RUN   TestShowRangesMultipleStores
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestShowRangesMultipleStores406550713
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    show_ranges_test.go:147: -- test log scope end --
test logs left over in: outputs.zip/logTestShowRangesMultipleStores406550713
--- FAIL: TestShowRangesMultipleStores (120.17s)
=== RUN   TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS
    sql_runner.go:121: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                                        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                                        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
            Error:          Received unexpected error:
                            query '
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:368
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                              | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                              |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:218
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:82
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                              | github.com/cockroachdb/cockroach/pkg/sql_test.TestShowRangesMultipleStores.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (2) query '
                              | SELECT DISTINCT
                              |         (
                              |         array_position(replica_localities, lease_holder_locality)
                              |         = array_position(replicas, lease_holder)
                              |         )
                              |     FROM [SHOW RANGES FROM DATABASE system WITH DETAILS]': expected:
                              | true
                              |
                              | got:
                              | true
                              | NULL
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    --- FAIL: TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS (55.83s)

Parameters:

attempt=1
race=true
run=1
shard=12
Help

Same failure on other branches

- #130556 sql: TestShowRangesMultipleStores failed [C-test-failure O-robot T-sql-queries branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

sql.TestShowRangesMultipleStores failed on master @ 83589fb87caa92fb42e83994f1691978f37e4cbb:

=== RUN   TestShowRangesMultipleStores
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestShowRangesMultipleStores3690225305
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    show_ranges_test.go:147: -- test log scope end --
test logs left over in: outputs.zip/logTestShowRangesMultipleStores3690225305
--- FAIL: TestShowRangesMultipleStores (113.51s)
=== RUN   TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS
    sql_runner.go:121: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                                        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                                        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
            Error:          Received unexpected error:
                            query '
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:368
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                              | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                              |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:218
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:82
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:365
                              | github.com/cockroachdb/cockroach/pkg/sql_test.TestShowRangesMultipleStores.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (2) query '
                              | SELECT DISTINCT
                              |         (
                              |         array_position(replica_localities, lease_holder_locality)
                              |         = array_position(replicas, lease_holder)
                              |         )
                              |     FROM [SHOW RANGES FROM DATABASE system WITH DETAILS]': expected:
                              | true
                              |
                              | got:
                              | true
                              | NULL
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    --- FAIL: TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS (48.60s)

Parameters:

attempt=1
race=true
run=3
shard=12
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 3 weeks ago

sql.TestShowRangesMultipleStores failed on master @ 8c52619671966cf15055f8b2e271ea82b59440c5:

=== RUN   TestShowRangesMultipleStores
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestShowRangesMultipleStores3725865541
    test_log_scope.go:81: use -show-logs to present logs inline
    test_server_shim.go:144: cluster virtualization disabled in global scope due to issue: #76378 (expected label: C-bug)
    show_ranges_test.go:147: -- test log scope end --
test logs left over in: outputs.zip/logTestShowRangesMultipleStores3725865541
--- FAIL: TestShowRangesMultipleStores (134.17s)
=== RUN   TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS
    sql_runner.go:121: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                                        github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:377
                                        github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
            Error:          Received unexpected error:
                            query '
                            (1) attached stack trace
                              -- stack trace:
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:380
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:76
                              | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                              |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:218
                              | github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError
                              |     github.com/cockroachdb/cockroach/pkg/testutils/soon.go:82
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).succeedsWithin
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:134
                              | github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).CheckQueryResultsRetry
                              |     github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:377
                              | github.com/cockroachdb/cockroach/pkg/sql_test.TestShowRangesMultipleStores.func1
                              |     github.com/cockroachdb/cockroach/pkg/sql_test/pkg/sql/show_ranges_test.go:137
                              | testing.tRunner
                              |     GOROOT/src/testing/testing.go:1689
                              | runtime.goexit
                              |     src/runtime/asm_amd64.s:1695
                            Wraps: (2) query '
                              | SELECT DISTINCT
                              |         (
                              |         array_position(replica_localities, lease_holder_locality)
                              |         = array_position(replicas, lease_holder)
                              |         )
                              |     FROM [SHOW RANGES FROM DATABASE system WITH DETAILS]': expected:
                              | true
                              |
                              | got:
                              | true
                              | NULL
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    --- FAIL: TestShowRangesMultipleStores/SHOW_RANGES_FROM_DATABASE_system_WITH_DETAILS (56.31s)

Parameters:

attempt=1
race=true
run=2
shard=12
Help

_{This test on roachdash | Improve this report!}

DrewKimball commented 2 weeks ago

It looks like we tried to fix this by adding a retry, since the test can fail depending on leaseholder configuration. I think it makes sense to also disable this test under race.

cockroach-teamcity commented 3 days ago

sql.TestShowRangesMultipleStores failed on master @ 9c3acd84e2fc639328b27a6cacd0e3893f1f90bb:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185a991]

Stack:

goroutine 187810 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00cff3868?, {0x8b9c428, 0xc009a4e600})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66c1600?, 0xce02b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00cff3988?, {0x8b9c428, 0xc009a4e840})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66c1600?, 0xce02b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8b9c428, 0xc009a4ee40}, 0x563)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00cff4768, {0x8b9c428?, 0xc009a4ee40?}, 0xc008199340?, 0x28?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc0072c4380, 0xc0072c43f0, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc013c8a008, {0x8b9c428, 0xc009a4e840}, 0xc007d9d448, {0x7fab219213b0, 0xc013e4f220})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x76b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8b9c428, 0xc009a4e840})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc002abc120, {0x8b9c428, 0xc009a4e840}, {0x7283f3b, 0x1e}, 0xc00bca5280)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc013c8a008, {0x8b9c3f0?, 0xc013810ff0?}, 0xc007d9d448, {0x7fab21921388, 0xc013e4f220})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc00637b180, {0x8bea810, 0xc013e4f220})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x32d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70b4b80?, 0xc00637b180}, {0x8bd3d18, 0xc01b1ba300})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0}, 0xc006811368, 0x76ad660)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0}, 0xc006811368?, 0xc0052fd7c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00e61e000?, {{0x470445?}, {0x8bf5f20?, 0xc00cc5db60?}}}, {0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0}, 0xc006811368, 0xc0052fd780)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7fab2398bc98?, 0x8b9c428?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc002abc120, {0x8b9c428, 0xc009a4e600}, {0xc00a2eeff0, 0x29}, 0xc005353aa0)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0}, 0xc006811368, 0xc0052fd740)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70b4b80, 0xc00637b180}, {0x8bd4f58, 0xc00c96e3c0}, 0xc006811368, 0xc013e4f210?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00a6ec000, {0x8bf6880, 0xc00bc029c0}, 0xc012cf97a0, 0xc015b5c360, 0xceffca0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc00a6ec000, {0x8bf6880, 0xc00bc029c0}, 0xc012cf97a0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 187776
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
run=4
shard=14
Help

_{This test on roachdash | Improve this report!}

github-actions[bot] commented 3 days ago

sql.TestShowRangesMultipleStores failed on master @ 6710d9ddd2d892807e01eefa24984fb0ed1553e6:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185aad1]

Stack:

goroutine 151804 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00ce3b868?, {0x8ba9ae8, 0xc00cfbbe30})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66ccc00?, 0xce14b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00ce3b988?, {0x8ba9ae8, 0xc00fc46390})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66ccc00?, 0xce14b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8ba9ae8, 0xc00fc46bd0}, 0x2a3f)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00ce3c768, {0x8ba9ae8?, 0xc00fc46bd0?}, 0xc014738420?, 0xe8?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0x2c, 0x20, 0xc00221dea0, 0xc00221df10, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00f26e008, {0x8ba9ae8, 0xc00fc46390}, 0xc011a4c488, {0x7f2d028cb4a0, 0xc00bd55870})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x76b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8ba9ae8, 0xc00fc46390})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc004d46bd0, {0x8ba9ae8, 0xc00fc46390}, {0x728fb0c, 0x1e}, 0xc003495280)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00f26e008, {0x8ba9ab0?, 0xc00e0eac30?}, 0xc011a4c488, {0x7f2d028cb478, 0xc00bd55870})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc01345ca80, {0x8bf7df0, 0xc00bd55870})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x32d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70c0160?, 0xc01345ca80}, {0x8be1300, 0xc0118f37a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0}, 0xc00ff5cf48, 0x76b94a0)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0}, 0xc00ff5cf48?, 0xc00317ff00)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00e15e000?, {{0x470445?}, {0x8c03500?, 0xc0131fe960?}}}, {0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0}, 0xc00ff5cf48, 0xc00317fe00)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f2d03c2b530?, 0x8ba9ae8?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc004d46bd0, {0x8ba9ae8, 0xc00cfbbe30}, {0xc007c158f0, 0x29}, 0xc0059deaa0)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0}, 0xc00ff5cf48, 0xc00317fd80)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70c0160, 0xc01345ca80}, {0x8be2538, 0xc008eba5a0}, 0xc00ff5cf48, 0xc00bd55860?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc009039a40, {0x8c03e60, 0xc012bef1e0}, 0xc00ff757a0, 0xc008c790e0, 0xcf11d80, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc009039a40, {0x8c03e60, 0xc012bef1e0}, 0xc00ff757a0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 151802
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
run=1
shard=14
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 3 days ago

sql.TestShowRangesMultipleStores failed with artifacts on master @ 6710d9ddd2d892807e01eefa24984fb0ed1553e6:

github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x40101d7948?, {0x82d4ac0, 0x402165bf50})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x68
panic({0x5e00080?, 0xc1e3390?})
    GOROOT/src/runtime/panic.go:770 +0x124
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x82d4ac0, 0x400b0fc090}, 0x563)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x4c
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0x40101d8740, {0x82d4ac0?, 0x400b0fc090?}, 0x4004a073aa?, 0x1a?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x70
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0x400b4fecb0, 0x400b4fed20, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x16f4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0x400bf58008, {0x82d4ac0, 0x402165bf50}, 0x400bf91448, {0xffff32ef3c08, 0x4019b9f610})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x538
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x82d4ac0, 0x402165bf50})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0xb0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x400a857cb0, {0x82d4ac0, 0x402165bf50}, {0x69c3bb2, 0x1e}, 0x4011aa3268)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0x400bf58008, {0x82d4a88?, 0x401680fb30?}, 0x400bf91448, {0xffff32ef3be0, 0x4019b9f610})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xc4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0x400545d340, {0x8323250, 0x4019b9f610})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x67f33c0?, 0x400545d340}, {0x830c688, 0x401471b3a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xdc
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30}, 0x4011fa0000, 0x6decb40)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x244
google.golang.org/grpc.getChainStreamHandler.func1({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30}, 0x4011fa0000?, 0x400bfd8100)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x78
google.golang.org/grpc.getChainStreamHandler.func1({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x4010840000?, {{0x1?}, {0x832e8a0?, 0x40165b8cc0?}}}, {0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30}, 0x4011fa0000, 0x400bfd80c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x2e4
google.golang.org/grpc.getChainStreamHandler.func1({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x40064ef9c8?, 0x82d4ac0?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x400a857cb0, {0x82d4ac0, 0x402165bdd0}, {0x4006dbef00, 0x29}, 0x40064efa88)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30}, 0x4011fa0000, 0x400bfd8080)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xc0
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x67f33c0, 0x400545d340}, {0x830d960, 0x400a296c30}, 0x4011fa0000, 0x4019b9f600?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x88
google.golang.org/grpc.(*Server).processStreamingRPC(0x40039a7c20, {0x832f200, 0x4004094340}, 0x400e3a2c60, 0x400f7b33b0, 0xc2e05a0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0xf50
google.golang.org/grpc.(*Server).handleStream(0x40039a7c20, {0x832f200, 0x4004094340}, 0x400e3a2c60, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x780
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 189073
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x14c

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

github-actions[bot] commented 3 days ago

sql.TestShowRangesMultipleStores failed on master @ b37ca45ae2eee82de92b778f4553f9fe8a19603e:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185aad1]

Stack:

goroutine 165386 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00d151868?, {0x8ba95e8, 0xc00d65fad0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66cc820?, 0xce14b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00d151988?, {0x8ba95e8, 0xc0089940c0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66cc820?, 0xce14b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8ba95e8, 0xc008994360}, 0x12fe)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00d152768, {0x8ba95e8?, 0xc008994360?}, 0xc00d1a9a20?, 0xe8?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0x18, 0xc, 0xc006f09dc0, 0xc006f09e30, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc008c19c08, {0x8ba95e8, 0xc0089940c0}, 0xc012dcdb08, {0x7ffb55652b00, 0xc00f3cecb0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x76b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8ba95e8, 0xc0089940c0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00e8b7050, {0x8ba95e8, 0xc0089940c0}, {0x728f6ac, 0x1e}, 0xc0068c9280)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc008c19c08, {0x8ba95b0?, 0xc001c047d0?}, 0xc012dcdb08, {0x7ffb55652ad8, 0xc00f3cecb0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc01513afc0, {0x8bf78f0, 0xc00f3cecb0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x32d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70bfd00?, 0xc01513afc0}, {0x8be0e00, 0xc00afd58c0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20}, 0xc007573950, 0x76b9040)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20}, 0xc007573950?, 0xc01305d640)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00c994000?, {{0x470445?}, {0x8c03000?, 0xc00f322180?}}}, {0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20}, 0xc007573950, 0xc01305d600)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7ffb5315ab28?, 0x8ba95e8?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00e8b7050, {0x8ba95e8, 0xc00d65fad0}, {0xc005c34780, 0x29}, 0xc008287aa0)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20}, 0xc007573950, 0xc01305d5c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70bfd00, 0xc01513afc0}, {0x8be2038, 0xc005141c20}, 0xc007573950, 0xc00f3ceca0?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc0054e6b40, {0x8c03960, 0xc00851a1a0}, 0xc0125927e0, 0xc001999560, 0xcf11d80, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc0054e6b40, {0x8c03960, 0xc00851a1a0}, 0xc0125927e0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 165412
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
run=1
shard=14
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 3 days ago

sql.TestShowRangesMultipleStores failed with artifacts on master @ 34429e118ba1aea868d29f252416bd83d7032375:

github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x40137c1948?, {0x82d3e60, 0x4006ed02a0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x68
panic({0x5dff7c0?, 0xc1e3390?})
    GOROOT/src/runtime/panic.go:770 +0x124
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x82d3e60, 0x4006ed0360}, 0x566)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x4c
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0x40137c2740, {0x82d3e60?, 0x4006ed0360?}, 0x400c0ea3e2?, 0x1a?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x70
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0x40203a6a10, 0x40203a6a80, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x16f4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0x4016bc2c08, {0x82d3e60, 0x4006ed02a0}, 0x400fb72008, {0xffff44656cd8, 0x400e8211a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x538
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x82d3e60, 0x4006ed02a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0xb0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4011578000, {0x82d3e60, 0x4006ed02a0}, {0x69c30d6, 0x1e}, 0x400d711268)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0x4016bc2c08, {0x82d3e28?, 0x401dace190?}, 0x400fb72008, {0xffff44656cb0, 0x400e8211a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xc4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0x400d963880, {0x8322570, 0x400e8211a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x294
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x67f2900?, 0x400d963880}, {0x830b9a8, 0x401a1451c0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xdc
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0}, 0x4016f5ae70, 0x6dec080)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x244
google.golang.org/grpc.getChainStreamHandler.func1({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0}, 0x4016f5ae70?, 0x4036b1f640)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x78
google.golang.org/grpc.getChainStreamHandler.func1({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x400a2ee000?, {{0x64594a0?}, {0x832dbc0?, 0x40051a2180?}}}, {0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0}, 0x4016f5ae70, 0x4036b1f600)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x2e4
google.golang.org/grpc.getChainStreamHandler.func1({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x400e6c49c8?, 0x82d3e60?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4011578000, {0x82d3e60, 0x4006ed0150}, {0x400bdec7e0, 0x29}, 0x400e6c4a88)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0}, 0x4016f5ae70, 0x4036b1f5c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xc0
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x67f2900, 0x400d963880}, {0x830cc80, 0x40115824b0}, 0x4016f5ae70, 0x400e821190?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x88
google.golang.org/grpc.(*Server).processStreamingRPC(0x4004134960, {0x832e520, 0x401cfdab60}, 0x400c2edd40, 0x40293f30e0, 0xc2e05a0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0xf50
google.golang.org/grpc.(*Server).handleStream(0x4004134960, {0x832e520, 0x401cfdab60}, 0x400c2edd40, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x780
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 213574
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x14c

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

sql.TestShowRangesMultipleStores failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x26fed37]

Stack:

goroutine 103070 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc008b14d00?, {0xe87b668, 0xc006c4bc80})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x74
panic({0xa674ca0?, 0x12ee6b70?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc008b14e38?, {0xe87b668, 0xc006c4bef0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x74
panic({0xa674ca0?, 0x12ee6b70?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0xe87b668, 0xc00e97ebd0}, 0x5e0)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x77
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc008b15fc0, {0xe87b668, 0xc00e97ebd0}, 0x1d, 0x1)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x14f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc006dbdc70, 0xc006dbdce0, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x3239
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00855e408, {0xe87b668, 0xc006c4bef0}, 0xc0085da008, {0x7f17175486e0, 0xc00c6f7820})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0xcbc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0xe87b668, 0xc006c4bef0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x106
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00b8f8b40, {0xe87b668, 0xc006c4bef0}, {0xb24c606, 0x1e}, 0xc007c52d00)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x247
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00855e408, {0xe87b630, 0xc0071a1770}, 0xc0085da008, {0x7f17175486b8, 0xc00c6f7820})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0x1c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc003448200, {0xe8c9450, 0xc00c6f7820})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x4f9
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0xb07aca0, 0xc003448200}, {0xe8b2960, 0xc0072393e0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xe6
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870}, 0xc00969af30, 0xb677cb8)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x449
google.golang.org/grpc.getChainStreamHandler.func1({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0x119
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870}, 0xc00969af30, 0xc006dece00)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0xd7
google.golang.org/grpc.getChainStreamHandler.func1({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0x119
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00854e000?, {{0xc006decdc0?}, {0xe8d4be0?, 0xc004c4d1f0?}}}, {0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870}, 0xc00969af30, 0xc006decdc0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x564
google.golang.org/grpc.getChainStreamHandler.func1({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0x119
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0xc00b8f8b40?, 0xe87b668?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x66
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00b8f8b40, {0xe87b668, 0xc006c4bc80}, {0xc006783650, 0x29}, 0xc00a8af778)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x247
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870}, 0xc00969af30, 0xc006decd80)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0x15c
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0xb07aca0, 0xc003448200}, {0xe8b3b50, 0xc006140870}, 0xc00969af30, 0xb677cb8)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0xdd
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00686d2c0, {0xe8d5540, 0xc00c5a1860}, 0xc0071eb680, 0xc001c3be30, 0x12fe4180, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x20fc
google.golang.org/grpc.(*Server).handleStream(0xc00686d2c0, {0xe8d5540, 0xc00c5a1860}, 0xc0071eb680, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0xfc5
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0x178
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 103062
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x265

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
race=true
run=2
shard=14
Help

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 2 days ago

sql.TestShowRangesMultipleStores failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185ad11]

Stack:

goroutine 165516 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00cf69820?, {0x8bb4548, 0xc00ec22390})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00cf69940?, {0x8bb4548, 0xc00ec224e0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8bb4548, 0xc005262b40}, 0x565)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00cf6a720, {0x8bb4548?, 0xc005262b40?}, 0xc00e049600?, 0x48?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc00452de30, 0xc00452dea0, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00d464008, {0x8bb4548, 0xc00ec224e0}, 0xc007effd48, {0x7f8154aa79e0, 0xc0068c6f80})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8bb4548, 0xc00ec224e0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006dee3f0, {0x8bb4548, 0xc00ec224e0}, {0x729925b, 0x1e}, 0xc002fd1268)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00d464008, {0x8bb4510?, 0xc00cbe0a50?}, 0xc007effd48, {0x7f8154aa79b8, 0xc0068c6f80})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc007289180, {0x8c027f0, 0xc0068c6f80})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70c9720?, 0xc007289180}, {0x8bebd00, 0xc01320f880})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0}, 0xc0094014e8, 0x76c2ec0)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0}, 0xc0094014e8?, 0xc007357b80)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00f500000?, {{0x470445?}, {0x8c0df00?, 0xc0104e2600?}}}, {0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0}, 0xc0094014e8, 0xc007357b40)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7f8155c7ea88?, 0x8bb4548?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc006dee3f0, {0x8bb4548, 0xc00ec22390}, {0xc00b5f02d0, 0x29}, 0xc00789daa0)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0}, 0xc0094014e8, 0xc007357b00)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70c9720, 0xc007289180}, {0x8becf38, 0xc00b6225a0}, 0xc0094014e8, 0xc0068c6ee0?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc009843c20, {0x8c0e860, 0xc013a371e0}, 0xc00fc5a120, 0xc0122270b0, 0xcf25d60, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc009843c20, {0x8c0e860, 0xc013a371e0}, 0xc00fc5a120, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 165512
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
run=14
shard=14
Help

Same failure on other branches

- #132991 sql: TestShowRangesMultipleStores failed [C-test-failure O-robot T-sql-queries branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

sql.TestShowRangesMultipleStores failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185ad11]

Stack:

goroutine 165394 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00a685820?, {0x8bb4548, 0xc010baa660})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc00a685940?, {0x8bb4548, 0xc010baa7b0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8bb4548, 0xc010baa9f0}, 0x565)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc00a686720, {0x8bb4548?, 0xc010baa9f0?}, 0xc016182840?, 0x48?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc0032f4bd0, 0xc0032f4c40, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc00ac40008, {0x8bb4548, 0xc010baa7b0}, 0xc00b3f0b48, {0x7fc9efc92dd0, 0xc00e565440})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8bb4548, 0xc010baa7b0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00d2f66c0, {0x8bb4548, 0xc010baa7b0}, {0x729925b, 0x1e}, 0xc0029ad268)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc00ac40008, {0x8bb4510?, 0xc014d89d10?}, 0xc00b3f0b48, {0x7fc9efc92da8, 0xc00e565440})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc00b2056c0, {0x8c027f0, 0xc00e565440})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70c9720?, 0xc00b2056c0}, {0x8bebd00, 0xc013439260})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0}, 0xc015287080, 0x76c2ec0)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0}, 0xc015287080?, 0xc0146710c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00bec6000?, {{0x470445?}, {0x8c0df00?, 0xc00a403620?}}}, {0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0}, 0xc015287080, 0xc014671080)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7fc9f043ff08?, 0x8bb4548?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00d2f66c0, {0x8bb4548, 0xc010baa660}, {0xc011784cf0, 0x29}, 0xc0078ceaa0)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0}, 0xc015287080, 0xc014671040)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70c9720, 0xc00b2056c0}, {0x8becf38, 0xc013cf60f0}, 0xc015287080, 0xc00e565430?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc011973860, {0x8c0e860, 0xc00f16cb60}, 0xc019d40ea0, 0xc014e369f0, 0xcf25d60, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc011973860, {0x8c0e860, 0xc00f16cb60}, 0xc019d40ea0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 165375
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
run=17
shard=14
Help

Same failure on other branches

- #132991 sql: TestShowRangesMultipleStores failed [C-test-failure O-robot T-sql-queries branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 23 hours ago

sql.TestShowRangesMultipleStores failed with artifacts on master @ b908a2f2bc3af2b1529b58e8242a37cfa6f6c1ca:

github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x400e2d78f8?, {0x82f1ba0, 0x401893bef0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x68
panic({0x5e158e0?, 0xc213390?})
    GOROOT/src/runtime/panic.go:770 +0x124
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x82f1ba0, 0x4008bec3f0}, 0x565)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x4c
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0x400e2d86f0, {0x82f1ba0?, 0x4008bec3f0?}, 0x400d042622?, 0x1a?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x70
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0x40061ab650, 0x40061ab6c0, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x16f4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0x400f246008, {0x82f1ba0, 0x401893bef0}, 0x400c65e908, {0xffff5d4d3ec8, 0x400e1943b0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x548
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x82f1ba0, 0x401893bef0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0xb0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4005ceb9e0, {0x82f1ba0, 0x401893bef0}, {0x69ddf44, 0x1e}, 0x400b1af248)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0x400f246008, {0x82f1b68?, 0x4013fcc690?}, 0x400c65e908, {0xffff5d4d3ea0, 0x400e1943b0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xc4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0x4009b95180, {0x83402b0, 0x400e1943b0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x28c
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x680c580?, 0x4009b95180}, {0x83296e8, 0x4012c6e6e0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xdc
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0}, 0x4009441998, 0x6e075a8)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x244
google.golang.org/grpc.getChainStreamHandler.func1({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0}, 0x4009441998?, 0x4006c2e840)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x78
google.golang.org/grpc.getChainStreamHandler.func1({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x4011b66000?, {{0x6471c80?}, {0x834b900?, 0x401201d560?}}}, {0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0}, 0x4009441998, 0x4006c2e800)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x2e4
google.golang.org/grpc.getChainStreamHandler.func1({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x40047fd9c8?, 0x82f1ba0?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4005ceb9e0, {0x82f1ba0, 0x40055dcf00}, {0x400c208c30, 0x29}, 0x40047fda88)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0}, 0x4009441998, 0x4006c2e7c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xc0
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x680c580, 0x4009b95180}, {0x832a9c0, 0x40096720f0}, 0x4009441998, 0x400e194300?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x88
google.golang.org/grpc.(*Server).processStreamingRPC(0x400c3921e0, {0x834c260, 0x4008fd01a0}, 0x40089d47e0, 0x40128cc9f0, 0xc310780, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0xf50
google.golang.org/grpc.(*Server).handleStream(0x400c3921e0, {0x834c260, 0x4008fd01a0}, 0x40089d47e0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x780
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 187703
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x14c

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #132991 sql: TestShowRangesMultipleStores failed [C-test-failure O-robot T-sql-queries branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 22 hours ago

sql.TestShowRangesMultipleStores failed with artifacts on master @ d50ca118e4b08725233e941e4e11a0f6bdc18c30:

github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x4014f338f8?, {0x82f1ba0, 0x401bdca240})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x68
panic({0x5e158e0?, 0xc213390?})
    GOROOT/src/runtime/panic.go:770 +0x124
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x82f1ba0, 0x401bdcac60}, 0x570)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x4c
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0x4014f346f0, {0x82f1ba0?, 0x401bdcac60?}, 0x401a7b4cf8?, 0x1a?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x70
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0x401c68e4d0, 0x401c68e5b0, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x16f4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0x40055ffc08, {0x82f1ba0, 0x401bdca240}, 0x400f6a3d48, {0xffff502d31d8, 0x401a588400})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x548
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x82f1ba0, 0x401bdca240})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0xb0
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4013430900, {0x82f1ba0, 0x401bdca240}, {0x69ddf44, 0x1e}, 0x400b147248)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0x40055ffc08, {0x82f1b68?, 0x40076bed20?}, 0x400f6a3d48, {0xffff502d31b0, 0x401a588400})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xc4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0x40072bb340, {0x83402b0, 0x401a588400})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x28c
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x680c580?, 0x40072bb340}, {0x83296e8, 0x40019ad8e0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xdc
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590}, 0x401ab9cd80, 0x6e075a8)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x244
google.golang.org/grpc.getChainStreamHandler.func1({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590}, 0x401ab9cd80?, 0x401dab7d80)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x78
google.golang.org/grpc.getChainStreamHandler.func1({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0x40065b2000?, {{0x6471c80?}, {0x834b900?, 0x40048c67e0?}}}, {0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590}, 0x401ab9cd80, 0x401dab7d40)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x2e4
google.golang.org/grpc.getChainStreamHandler.func1({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xa0
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x400fa009c8?, 0x82f1ba0?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x34
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4013430900, {0x82f1ba0, 0x401bdca090}, {0x4014c7dce0, 0x29}, 0x400fa00a88)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x100
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590}, 0x401ab9cd80, 0x401dab7d00)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xc0
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x680c580, 0x40072bb340}, {0x832a9c0, 0x4002685590}, 0x401ab9cd80, 0x401a5883f0?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x88
google.golang.org/grpc.(*Server).processStreamingRPC(0x4016f4c1e0, {0x834c260, 0x4013e66b60}, 0x4008f7d440, 0x401b5c2c60, 0xc310780, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0xf50
google.golang.org/grpc.(*Server).handleStream(0x4016f4c1e0, {0x834c260, 0x4013e66b60}, 0x4008f7d440, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x780
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xb4
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 182158
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x14c

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #132991 sql: TestShowRangesMultipleStores failed [C-test-failure O-robot T-sql-queries branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 17 hours ago

sql.TestShowRangesMultipleStores failed on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

Fatal error:

panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference [recovered]
    panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x185ad11]

Stack:

goroutine 153954 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc01605b820?, {0x8bb4548, 0xc01d5864b0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc01605b940?, {0x8bb4548, 0xc01d586600})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:226 +0x65
panic({0x66d4500?, 0xce28b40?})
    GOROOT/src/runtime/panic.go:770 +0x132
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit(0x0, {0x8bb4548, 0xc01d586c90}, 0x570)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 +0x71
github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace(0xc01605c720, {0x8bb4548?, 0xc01d586c90?}, 0xc005f5b340?, 0x48?)
    github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 +0x6d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive(_, {_, _}, _, {_, _}, {{0xe, 0x2, 0xc004f5d2d0, 0xc004f5d340, ...}, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 +0x1f9f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot(0xc010bb7008, {0x8bb4548, 0xc01d586600}, 0xc004717208, {0x7ffa1d0769a0, 0xc001eab210})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 +0x797
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1({0x8bb4548, 0xc01d586600})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 +0x85
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0070b7d40, {0x8bb4548, 0xc01d586600}, {0x729925b, 0x1e}, 0xc006853268)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot(0xc010bb7008, {0x8bb4510?, 0xc00908e870?}, 0xc004717208, {0x7ffa1d076978, 0xc001eab210})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 +0xf6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot(0xc010ab2700, {0x8c027f0, 0xc001eab210})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 +0x30d
github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler({0x70c9720?, 0xc010ab2700}, {0x8bebd00, 0xc0053fc440})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 +0xd8
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40}, 0xc006faca50, 0x76c2ec0)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 +0x35b
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40}, 0xc006faca50?, 0xc00c7179c0)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 +0x6c
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor({0xc00d218000?, {{0x470445?}, {0x8c0df00?, 0xc00ed72780?}}}, {0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40}, 0xc006faca50, 0xc00c717980)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 +0x384
google.golang.org/grpc.getChainStreamHandler.func1({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1({0x7ffa1e5b60a0?, 0x8bb4548?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 +0x27
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0070b7d40, {0x8bb4548, 0xc01d5864b0}, {0xc010f97050, 0x29}, 0xc00dfc9aa0)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 +0x15d
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40}, 0xc006faca50, 0xc00c717940)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 +0xdf
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x70c9720, 0xc010ab2700}, {0x8becf38, 0xc00bb56b40}, 0xc006faca50, 0xc001eab200?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00d1a6f00, {0x8c0e860, 0xc005805860}, 0xc018591320, 0xc00d5a68d0, 0xcf25d60, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 +0x1247
google.golang.org/grpc.(*Server).handleStream(0xc00d1a6f00, {0x8c0e860, 0xc005805860}, 0xc018591320, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 +0x999
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 153915
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x136

Log preceding fatal error

``` * | GOROOT/src/runtime/signal_unix.go:881 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotQueue).Admit * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:165 * | github.com/cockroachdb/cockroach/pkg/util/admission.(*SnapshotPacer).Pace * | github.com/cockroachdb/cockroach/pkg/util/admission/snapshot_queue.go:292 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*kvBatchSnapshotStrategy).Receive * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:918 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).receiveSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_snapshot.go:1693 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot.func1 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:209 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).HandleSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:206 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*RaftTransport).RaftSnapshot * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/raft_transport.go:638 * | github.com/cockroachdb/cockroach/pkg/kv/kvserver._MultiRaft_RaftSnapshot_Handler * | github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:267 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.StreamServerInterceptor.func13 * | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:160 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func4 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:174 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.streamInterceptor * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:152 * | google.golang.org/grpc.getChainStreamHandler.func1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1481 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2.1 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:141 * | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr * | github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:349 * | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func2 * | github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:140 * | google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1472 * | google.golang.org/grpc.(*Server).processStreamingRPC * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1636 * | google.golang.org/grpc.(*Server).handleStream * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:1716 * | google.golang.org/grpc.(*Server).serveStreams.func1.1 * | google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 * | runtime.goexit * | src/runtime/asm_amd64.s:1695 * Wraps: (2) runtime error: invalid memory address or nil pointer dereference * Error types: (1) *withstack.withStack (2) runtime.errorString * ```

Parameters:

attempt=1
run=6
shard=14
Help

Same failure on other branches

- #132991 sql: TestShowRangesMultipleStores failed [C-test-failure O-robot T-sql-queries branch-release-24.3 release-blocker]

This test on roachdash | Improve this report!

msbutler commented 15 hours ago

just hit this in ci: https://mesolite.cluster.engflow.com/invocations/default/564b0ac6-3be2-4507-8145-ad80db59a7dc

@DrewKimball is there a hot fix for this, or can we skip this test for now?

cockroachdb / cockroach

sql: TestShowRangesMultipleStores failed #130490