sql/tests: TestRandomSyntaxFunctions failed

cockroach-teamcity commented 2 months ago

sql/tests.TestRandomSyntaxFunctions failed with artifacts on release-23.2 @ 06ff271990ec79047fb4739068e98449ad191e0b:

github.com/cockroachdb/pebble.(*scanInternalIterator).seekGE(...)
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/scan_internal.go:909
github.com/cockroachdb/pebble.scanInternalImpl({0x7c152c0?, 0xbc46ca0?}, {0xc03877c608?, 0xc01b057015?, 0xe?}, {0xc03877c630?, 0x0?, 0xc01b057007?}, 0xc00472af00, 0xc004f169d8)
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/scan_internal.go:652 +0x16c
github.com/cockroachdb/pebble.(*DB).ScanStatistics(_, {_, _}, {_, _, _}, {_, _, _}, {0x989680})
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:2900 +0x397
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).ScanStorageInternalKeys(0xc0014d6500?, {0xc03877c608?, 0x5?, 0x8?}, {0xc03877c630?, 0x3?, 0x8?}, 0xa?)
    github.com/cockroachdb/cockroach/pkg/storage/pebble.go:2403 +0x69
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.ScanStorageInternalKeys.func1({0x11c8b20?, 0xc044315c38?}, 0xc00480b278?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:184 +0x56
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.execStoreCommand.func1({0x7c15410?, 0xc025251500?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:46 +0x28
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc044315c20, {0x7c15410?, 0xc025251500}, {0x0?, 0xc00480b368?}, 0xc00480b350)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.execStoreCommand({0x0?}, {0x7c15410, 0xc025251500}, {0x0?, 0x0?}, 0xc00480b3b0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:45 +0x8a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.ScanStorageInternalKeys({0x0?}, {0x7c15410, 0xc025251500}, 0xc02a375200)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:182 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver._PerStore_ScanStorageInternalKeys_Handler.func1({0x7c15410, 0xc025251500}, {0x6221740?, 0xc02a375200})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:621 +0x72
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200}, 0xc054ba19a0, 0xc061938a98)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:116 +0x2b7
google.golang.org/grpc.getChainUnaryHandler.func1({0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200}, 0xc054ba19a0?, 0xc02a375300)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
google.golang.org/grpc.getChainUnaryHandler.func1({0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc002474000?, {{0x426e05?}, {0x7c535e0?, 0xc04a6d4b10?}}}, {0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200}, 0xc054ba19a0, 0xc02a375280)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x273
google.golang.org/grpc.getChainUnaryHandler.func1({0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x7c15410?, 0xc0252514a0?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc044315c20, {0x7c15410?, 0xc0252514a0}, {0x0?, 0xc002370968?}, 0xc002370960)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x7c15410?, 0xc0252514a0?}, {0x6221740?, 0xc02a375200?}, 0xc054ba19a0?, 0xc061938a98?)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x7c15410, 0xc0252514a0}, {0x6221740, 0xc02a375200}, 0xc002370a38?, 0x5b0a8c0?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1145 +0x85
github.com/cockroachdb/cockroach/pkg/kv/kvserver._PerStore_ScanStorageInternalKeys_Handler({0x6182420?, 0xc021008b80}, {0x7c15410, 0xc0252514a0}, 0xc038260f50, 0xc04962ef40)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:623 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0014c8000, {0x7c67f20, 0xc00e324b60}, 0xc056820900, 0xc046ddd170, 0xb5109b0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1335 +0xde7
google.golang.org/grpc.(*Server).handleStream(0xc0014c8000, {0x7c67f20, 0xc00e324b60}, 0xc056820900, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1712 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 3998470
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x145

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #130065 sql/tests: TestRandomSyntaxFunctions failed [A-storage C-test-failure O-robot T-storage branch-release-24.1 release-blocker] - #130002 sql/tests: TestRandomSyntaxFunctions failed [A-storage C-test-failure O-robot T-storage branch-release-24.2 release-blocker] - #129952 sql/tests: TestRandomSyntaxFunctions failed [A-storage C-test-failure O-robot T-storage branch-release-24.2.1-rc]

/cc @cockroachdb/sql-foundations

This test on roachdash | Improve this report!

Jira issue: CRDB-41965

cockroach-teamcity commented 2 months ago

sql/tests.TestRandomSyntaxFunctions failed with artifacts on release-23.2 @ 74d8d18a9ac3fbda5c1feb6155db8ff464a3531e:

github.com/cockroachdb/pebble.(*scanInternalIterator).seekGE(...)
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/scan_internal.go:909
github.com/cockroachdb/pebble.scanInternalImpl({0x7c15560?, 0xbc47ca0?}, {0xc01bbd2e38?, 0xc012f26007?, 0xa?}, {0xc01bbd2e60?, 0x0?, 0x0?}, 0xc00930fc20, 0xc00949c9d8)
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/scan_internal.go:652 +0x16c
github.com/cockroachdb/pebble.(*DB).ScanStatistics(_, {_, _}, {_, _, _}, {_, _, _}, {0x989680})
    github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:2900 +0x397
github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).ScanStorageInternalKeys(0xc0024ad180?, {0xc01bbd2e38?, 0x2?, 0x8?}, {0xc01bbd2e60?, 0x6?, 0x8?}, 0xa?)
    github.com/cockroachdb/cockroach/pkg/storage/pebble.go:2403 +0x69
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.ScanStorageInternalKeys.func1({0x11c8b20?, 0xc001341c38?}, 0xc00a66d278?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:184 +0x56
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.execStoreCommand.func1({0x7c156b0?, 0xc026df7680?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:46 +0x28
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001341c20, {0x7c156b0?, 0xc026df7680}, {0x0?, 0xc00a66d368?}, 0xc00a66d350)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.execStoreCommand({0x0?}, {0x7c156b0, 0xc026df7680}, {0x0?, 0x0?}, 0xc00a66d3b0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:45 +0x8a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.Server.ScanStorageInternalKeys({0x0?}, {0x7c156b0, 0xc026df7680}, 0xc0062f3700)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores_server.go:182 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver._PerStore_ScanStorageInternalKeys_Handler.func1({0x7c156b0, 0xc026df7680}, {0x62219e0?, 0xc0062f3700})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:621 +0x72
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700}, 0xc03e948ae0, 0xc010d5bab8)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:116 +0x2b7
google.golang.org/grpc.getChainUnaryHandler.func1({0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700}, 0xc03e948ae0?, 0xc0062f3840)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
google.golang.org/grpc.getChainUnaryHandler.func1({0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0024e2000?, {{0x426e05?}, {0x7c53880?, 0xc048f54390?}}}, {0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700}, 0xc03e948ae0, 0xc0062f3780)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x273
google.golang.org/grpc.getChainUnaryHandler.func1({0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700})
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xb2
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x7c156b0?, 0xc026df7620?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001341c20, {0x7c156b0?, 0xc026df7620}, {0x0?, 0xc006d4e968?}, 0xc006d4e960)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x7c156b0?, 0xc026df7620?}, {0x62219e0?, 0xc0062f3700?}, 0xc03e948ae0?, 0xc010d5bab8?)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x7c156b0, 0xc026df7620}, {0x62219e0, 0xc0062f3700}, 0xc006d4ea38?, 0x5b0ab40?)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1145 +0x85
github.com/cockroachdb/cockroach/pkg/kv/kvserver._PerStore_ScanStorageInternalKeys_Handler({0x61826c0?, 0xc002e28080}, {0x7c156b0, 0xc026df7620}, 0xc01583ae00, 0xc0070e51e0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/bazel-out/k8-fastbuild/bin/pkg/kv/kvserver/kvserver_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvserver/storage_services.pb.go:623 +0x135
google.golang.org/grpc.(*Server).processUnaryRPC(0xc002bcf2c0, {0x7c681c0, 0xc0119691e0}, 0xc00a72a5a0, 0xc001827110, 0xb5119d0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1335 +0xde7
google.golang.org/grpc.(*Server).handleStream(0xc002bcf2c0, {0x7c681c0, 0xc0119691e0}, 0xc00a72a5a0, 0x0)
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:1712 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.1()
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:947 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 3602414
    google.golang.org/grpc/external/org_golang_google_grpc/server.go:958 +0x145

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #130065 sql/tests: TestRandomSyntaxFunctions failed [A-storage C-test-failure O-robot T-storage branch-release-24.1 release-blocker] - #130002 sql/tests: TestRandomSyntaxFunctions failed [A-storage C-test-failure O-robot T-storage branch-release-24.2 release-blocker] - #129952 sql/tests: TestRandomSyntaxFunctions failed [A-storage C-test-failure O-robot T-storage branch-release-24.2.1-rc]

This test on roachdash | Improve this report!

fqazi commented 2 months ago

We saw a failure inside Pebble again on this test. Can someone on storage take a look?

RaduBerinde commented 2 months ago

Need to manually backport https://github.com/cockroachdb/cockroach/pull/130953 to 23.2.

RaduBerinde commented 2 months ago

Fixed in #131694.

cockroachdb / cockroach

sql/tests: TestRandomSyntaxFunctions failed #130275