kvserver: warning about empty admissison headers

[stevendanna]

Describe the problem

When reading through logs I've been seeing this:

W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281  empty admission header provided by goroutine 3952 [running]:                                                                                                       
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +runtime/debug.Stack()                                                                                                                                              
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      GOROOT/src/runtime/debug/stack.go:24 +0x64                                                                                                                   
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0x1400107e280, {0x1082fdc88, 0x14008ca4a20}, {0x1082d2f90, 0x14008
63e708}, {0x1, {0x0, 0x0, 0x0}, {0x0, ...}, ...})                                                                                                                                                                                                            
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1029 +0x234                                                               
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntents(...)                                                              
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:997                                       
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntents(0x1400244ad58?, {0x1082fdc88, 0x14008ca4a20}, {0x0, 0x0, 0x0, 0x0,
 0x0}, {0x14005e90f00, 0x1, ...}, ...)                                                                                                                                                                                                                       
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:595 +0x2b0                                                                
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1.1({0x1082fdc88?, 0x14008ca4a20?})                      
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:517 +0x64                                                                 
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x1082fdcc0?, 0x14008bb4120?}, {0x10603b978, 0x17}, 0x6fc23ac00, 0x1400244aec0)                 
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x90                                                                                       
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1({0x1082fdcc0, 0x14008bb4120})                          
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:515 +0xb8                                                                 
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()                                                                                   
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x128                                                                                         
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx                                                                                
W231019 09:51:14.423104 3952 kv/kvserver/intentresolver/intent_resolver.go:1029  [-] 281 +      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x33c           

Is this something unexpected that we should be fixing? I imagine the goal of this log was to identify places that might benefit from admission headers.

Jira issue: CRDB-32550

[sumeerbhola]

What test were you running? We'll need to hack up some extra instrumentation to figure out where the call originated.

[aadityasondhi]

Just saw this in the wild, in the logs for https://github.com/cockroachdb/cockroach/issues/112373. Will take a closer look in a bit.

W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172  empty admission header provided by ‹goroutine 409266 [running]:›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹runtime/debug.Stack()›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ GOROOT/src/runtime/debug/stack.go:24 +0x65›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc000ed1540, {0x79471f8, 0xc01e69ae40}, {0x7917110, 0xc00cadeb60}, {0x1, {0x0, 0x0, 0x0}, {0x0, ...}, ...})›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1029 +0x2ed›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntent(_, {_, _}, {{{0xc008b44cd0, 0x9, 0x10}, {0x0, 0x0, 0x0}}, {{0x30, ...}, ...}, ...}, ...)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:983 +0xfc›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc0009b1270, {0x79471f8, 0xc01e69ae40}, {0xc08495a6c0, {0x178df1f1ceea0fd5, 0x2, 0x0}, 0x0, 0x0, 0x1, ...}, ...)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:633 +0x8c6›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc0009b1270, {0x79471f8, 0xc01e69ae40}, {0xc08495a6c0, {0x178df1f1ceea0fd5, 0x2, 0x0}, 0x0, 0x0, 0x1, ...}, ...)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:406 +0x618›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc0009b1220, {0x79471f8, 0xc01e69ae40}, 0xc070fd8900, 0xc0085bf4f0?)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:348 +0xa0f›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc0085bf720?, {0x79471f8, 0xc01e69ae40}, 0xc070fd8900?, {0xc08495a6c0, {0x178df1f1ceea0fd5, 0x2, 0x0}, 0x0, 0x0, ...}, ...)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239 +0x2ec›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc008e8f900, {0x79471f8, 0xc01e69ae40}, 0xc08495a5a0, 0x66b2a50)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x34c›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc008e8f900, {0x79471f8?, 0xc01e69ae10?}, 0xc08495a5a0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x6e9›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc001b7b500, {0x79471f8?, 0xc01e69adb0?}, 0xc08495a5a0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x885›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x1?, {0x79471f8, 0xc01e69adb0}, 0xc08495a5a0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x10f›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc001da3000, {0x79471f8?, 0xc01e69ad80?}, {0xc0020f1260?}, 0xc08495a5a0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/server/node.go:1328 +0x55b›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc001da3000, {0x79471f8, 0xc01e69acc0}, 0xc08495a5a0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/server/node.go:1459 +0x2d5›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x79471f8, 0xc01e69acc0}, {0x6236560?, 0xc08495a5a0})›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10410 +0x78›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ServerInterceptor.func1({0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0}, 0xc02180dcc0, 0xc024ca9a28)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x254›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0})›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0}, 0xc02180dcc0?, 0xc026f9a5c0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x83›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0})›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc001f00000?, {{0x46f147?}, {0x7985520?, 0xc0007379e0?}}}, {0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0}, 0xc02180dcc0, 0xc026f9a540)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x288›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0})›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb9›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x79471f8?, 0xc01e69acc0?})›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x3c›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001ed5320, {0x79471f8, 0xc01e69acc0}, {0x0?, 0xc0085c1980?}, 0xc0085c1978)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd1›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x79471f8?, 0xc01e69acc0?}, {0x6236560?, 0xc08495a5a0?}, 0xc02180dcc0?, 0xc024ca9a28?)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x95›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.chainUnaryInterceptors.func1({0x79471f8, 0xc01e69acc0}, {0x6236560, 0xc08495a5a0}, 0xc003855a50?, 0x5976600?)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x8f›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x6207700?, 0xc001da3000}, {0x79471f8, 0xc01e69acc0}, 0xc0278a09a0, 0xc001fa2300)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10412 +0x138›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.(*Server).processUnaryRPC(0xc00066b680, {0x79993c0, 0xc026135d40}, 0xc0342dab40, 0xc000c18c90, 0xb03f120, 0x0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd33›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.(*Server).handleStream(0xc00066b680, {0x79993c0, 0xc026135d40}, 0xc0342dab40, 0x0)›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0xa36›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹google.golang.org/grpc.(*Server).serveStreams.func1.2()›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x98›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹created by google.golang.org/grpc.(*Server).serveStreams.func1›
W231014 10:17:35.820948 409266 kv/kvserver/intentresolver/intent_resolver.go:1029 ⋮ [T1,Vsystem,n3,s3,r87/3:‹/Table/11{1/1-2}›] 1172 +‹ google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x28a›

[erikgrinaker]

Should we mark this as a GA blocker?

[aadityasondhi]

Should we mark this as a GA blocker?

I don't think it is one, unless we see some performance consequence, the internal scale testing would surface these.

[aadityasondhi]

Added some additional logging and found one instance of where we missed the admission header:

E231107 21:15:51.925434 57687 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r2/1:/System/NodeLiveness{-Max}] 2823  empty admission header provided by RequestLease [/System/NodeLiveness,/Min)
E231107 21:15:51.925443 57671 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r1/1:/{Min-System/NodeL…}] 2824  empty admission header provided by RequestLease [/Min,/Min)
I231107 21:15:51.925596 1 server/node.go:662  [T1,Vsystem,n1] 2825  initialized store s1
I231107 21:15:51.925616 1 kv/kvserver/stores.go:263  [T1,Vsystem,n1] 2826  read 0 node addresses from persistent storage
I231107 21:15:51.925642 1 server/node.go:756  [T1,Vsystem,n1] 2827  started with engine type 2
I231107 21:15:51.925649 1 server/node.go:758  [T1,Vsystem,n1] 2828  started with attributes []
I231107 21:15:51.925710 1 1@server/server.go:1980  [T1,Vsystem,n1] 2829  starting https server at 127.0.0.1:52685 (use: 127.0.0.1:52685)
I231107 21:15:51.925720 1 1@server/server.go:1985  [T1,Vsystem,n1] 2830  starting postgres server at 127.0.0.1:52687 (use: 127.0.0.1:52687)
I231107 21:15:51.925727 1 1@server/server.go:1988  [T1,Vsystem,n1] 2831  starting grpc server at 127.0.0.1:52686
I231107 21:15:51.925734 1 1@server/server.go:1989  [T1,Vsystem,n1] 2832  advertising CockroachDB node at 127.0.0.1:52686
I231107 21:15:51.925734 57756 2@kv/kvserver/replica_proposal.go:585  [T1,Vsystem,n1,s1,r2/1:/System/NodeLiveness{-Max},raft] 2833  acquired system range lease: repl=(n1,s1):1 seq=1 start=0,0 exp=1699391757.925409000,0 pro=1699391751.925409000,0
I231107 21:15:51.925786 57755 2@kv/kvserver/replica_proposal.go:585  [T1,Vsystem,n1,s1,r1/1:/{Min-System/NodeL…},raft] 2834  acquired system range lease: repl=(n1,s1):1 seq=1 start=0,0 exp=1699391757.925418000,1 pro=1699391751.925418000,1
E231107 21:15:51.926245 57692 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r7/1:/Table/{3-4}] 2835  empty admission header provided by RequestLease [/Table/3,/Min)
E231107 21:15:51.926485 57763 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r9/1:/Table/{5-6}] 2836  empty admission header provided by RequestLease [/Table/5,/Min)
E231107 21:15:51.926489 57454 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r27/1:/Table/2{4-5}] 2837  empty admission header provided by RequestLease [/Table/24,/Min)
E231107 21:15:51.927064 57673 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r14/1:/Table/1{1-2}] 2838  empty admission header provided by RequestLease [/Table/11,/Min)
E231107 21:15:51.929512 57892 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r3/1:/System/{NodeLive…-tsd}] 2839  empty admission header provided by RequestLease [/System/NodeLivenessMax,/Min)
I231107 21:15:51.929907 57837 3@pebble/event.go:752  [n1,s1,pebble] 2840  [JOB 5] WAL created 000004
I231107 21:15:51.929948 57087 kv/kvclient/rangefeed/rangefeedcache/watcher.go:340  [T1,Vsystem,n1] 2841  spanconfig-subscriber: established range feed cache
I231107 21:15:51.930147 1 1@util/log/event_log.go:32  [T1,Vsystem,n1] 2842 ={"Timestamp":1699391751930146000,"EventType":"node_join","NodeID":1,"StartedAt":1699391751920133000,"LastUp":1699391751920133000}
I231107 21:15:51.930225 57954 kv/kvclient/rangefeed/rangefeedcache/watcher.go:340  [T1,Vsystem,n1] 2843  settings-watcher: established range feed cache
E231107 21:15:51.930451 57765 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r4/1:/System{/tsd-tse}] 2844  empty admission header provided by RequestLease [/System/tsd,/Min)
I231107 21:15:51.934931 57862 3@pebble/event.go:752  [n1,s1,pebble] 2845  [JOB 6] WAL created 000005
E231107 21:15:51.936083 57924 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r23/1:/Table/2{0-1}] 2846  empty admission header provided by RequestLease [/Table/20,/Min)
E231107 21:15:51.936484 57940 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r50/1:/Table/4{7-8}] 2847  empty admission header provided by RequestLease [/Table/47,/Min)
E231107 21:15:51.936922 57926 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r10/1:/Table/{6-7}] 2848  empty admission header provided by RequestLease [/Table/6,/Min)
E231107 21:15:51.936939 57963 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r5/1:/{Systemtse-Table/0}] 2849  empty admission header provided by RequestLease [/System/"tse",/Min)
E231107 21:15:51.937400 57681 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r6/1:/Table/{0-3}] 2850  empty admission header provided by RequestLease [/Table/0,/Min)
I231107 21:15:51.937488 57794 kv/kvserver/queue.go:612  [T1,Vsystem,n1,s1,r6/1:/Table/{0-3},raft] 2851  rate limited in MaybeAdd (replicate): throttled on async limiting semaphore
I231107 21:15:51.937931 1 sql/sqlliveness/slinstance/slinstance.go:434  [T1,Vsystem,n1] 2852  starting SQL liveness instance
E231107 21:15:51.938259 58097 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r42/1:/Table/{39-40}] 2853  empty admission header provided by RequestLease [/Table/39,/Min)
E231107 21:15:51.938474 58117 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r8/1:/Table/{4-5}] 2854  empty admission header provided by RequestLease [/Table/4,/Min)
E231107 21:15:51.938526 58036 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r34/1:/{NamespaceTab…-Table/32}] 2855  empty admission header provided by RequestLease [/NamespaceTable/Max,/Min)
E231107 21:15:51.938582 58133 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r15/1:/Table/1{2-3}] 2856  empty admission header provided by RequestLease [/Table/12,/Min)
I231107 21:15:51.938587 57774 sql/sqlliveness/slstorage/slstorage.go:543  [T1,Vsystem,n1] 2857  inserted sqlliveness session 010180d741b9438ef7448795d6329e5e8166bc with expiry 1699391791.937958000,0
I231107 21:15:51.938629 57774 sql/sqlliveness/slinstance/slinstance.go:258  [T1,Vsystem,n1] 2858  created new SQL liveness session 010180d741b9438ef7448795d6329e5e8166bc
I231107 21:15:51.938648 1 sql/sqlinstance/instancestorage/instancestorage.go:334  [T1,Vsystem,n1] 2859  assigning instance id to rpc addr 127.0.0.1:52686 and sql addr 127.0.0.1:52687
E231107 21:15:51.938723 58135 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r49/1:/Table/4{6-7}] 2860  empty admission header provided by RequestLease [/Table/46,/Min)
E231107 21:15:51.938907 58078 kv/kvserver/replica_send.go:122  [T1,Vsystem,n1,s1,r18/1:/Table/1{5-6}] 2861  empty admission header provided by RequestLease [/Table/15,/Min)
I231107 21:15:51.939015 1 server/server_sql.go:1537  [T1,Vsystem,n1] 2862  bound sqlinstance: Instance{RegionPrefix: gA==, InstanceID: 1, SQLAddr: 127.0.0.1:52687, RPCAddr: 127.0.0.1:52686, SessionID: 010180d741b9438ef7448795d6329e5e8166bc, Locality: , BinaryVersion: 1000023.2-2}

The RequestLease batch request seems like a culprit.

Going to additional logging and fix these all in a single batch.

[erikgrinaker]

Should we mark this as a GA blocker?

I don't think it is one, unless we see some performance consequence, the internal scale testing would surface these.

Yeah, probably isn't a functional problem, but operators are jumpy about backtraces in logs.

[blathers-crl[bot]]

Hi @aadityasondhi, please add branch-* labels to identify which branch(es) this release-blocker affects.

_{:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[aadityasondhi]

Ack. Added the labels, but I am actively working on cleaning this up.

[itsbilal]

We just observed this in the CCT test cluster as well, shortly after upgrading to 23.2 beta 1 and dropping a job holding the protected timestamp back.

[aadityasondhi]

FYI: I have a draft WIP PR to track these down, https://github.com/cockroachdb/cockroach/pull/114077.

Based on an internal conversation[1], I will remove this warning message and the included stack trace for 23.2 since it does not point to the source anyways since this is happening in an async task. Will continue to track these down in master.

[1] https://cockroachlabs.slack.com/archives/C067NG669KJ/p1701876295435589

[aadityasondhi]

With https://github.com/cockroachdb/cockroach/pull/116286 merged and backported, I am closing this out. If we see more of those warnings in tests we can look at them separately. It is no longer a user facing warning after this:https://github.com/cockroachdb/cockroach/pull/115705 so it should no longer block the release.

[renatolabs]

Correct me if I'm wrong, but I believe we're still seeing empty admission header with the same stack as originally reported here even after #116286 was merged. I happened to notice it while looking at something else over in #116681.

W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216  empty admission header provided by ‹goroutine 3884 [running]:›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹runtime/debug.Stack()›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    GOROOT/src/runtime/debug/stack.go:24 +0x5e›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc001809180, {0x7e95b60, 0xc00194ed90}, {0x7e611a8, 0xc006130870}, {0x1, {0x0, 0x0, 0x0}, {0x0, ...}, ...})›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1032 +0x345›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntents(...)›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:998›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntents(0xc001809180?, {0x7e95b60, 0xc00194ed90}, {0x0, 0x0, 0x0, 0x0, 0x0}, {0xc005f36f00, 0x1, ...}, ...)›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:596 +0x4b1›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1.1({0x7e95b60, 0xc00194ed90?})›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:518 +0x99›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x7e95af0?, 0xc005fe29f0?}, {0x69a4ae2, 0x17}, 0x6fc23ac00, 0xc0033b5ec8)›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0xab›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1({0x7e95af0, 0xc005fe29f0})›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:516 +0xf1›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x159›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 3883›
W231218 17:45:56.834439 3884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 216 +‹    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x50b›

[yuzefovich]

I too still see this on master on 5f76d93b2ad8d29c14c9b176753ec33cbf81e3fe:

W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439  empty admission header provided by ‹goroutine 219884 [running]:›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹runtime/debug.Stack()›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  GOROOT/src/runtime/debug/stack.go:24 +0x67›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc003111b80, {0xd1718b8, 0xc017da4b90}, {0xd1444c0, 0xc00db62820}, {0x1, {0x0, 0x0, 0x0}, {0x0, ...}, ...})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1032 +0x4dd›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntent(_, {_, _}, {{{0xc004522028, 0x2, 0x8}, {0x0, 0x0, 0x0}}, {{0xd4, ...}, ...}, ...}, ...)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:984 +0x127›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc005af9220, {0xd1718b8, 0xc017da4b90}, {0x0, {0x17a2352cd5ac5263, 0x0, 0x0}, 0x0, 0x0, 0x0, ...}, ...)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:662 +0xd68›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn.func3({0xd1718f0, 0xc0079de900})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:401 +0x365›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc005af9220, {0xd1718f0, 0xc0079de900}, {0x0, {0x17a2352cd5ac5263, 0x0, 0x0}, 0x0, 0x0, 0x0, ...}, ...)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:447 +0x7db›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc005af91d0, {0xd1718f0, 0xc0079de900}, 0xc003849b00, 0xc012653a50?)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:350 +0x103a›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc030049010?, {0xd1718f0, 0xc0079de900}, 0xc003849b00, {0x0, {0x17a2352cd5ac5263, 0x0, 0x0}, 0x0, 0x0, ...}, ...)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241 +0x329›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc0068c8000, {0xd1718f0, 0xc0079de900}, 0xc00537a000, 0x9d839a8)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x4d7›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc0068c8000, {0xd1718f0, 0xc0079de8d0}, 0xc00537a000)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x5a5›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc005d8aa80, {0xd1718f0, 0xc0079de8a0}, 0xc00537a000)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0xd32›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc029b8e378?, {0xd1718f0, 0xc0079de8a0}, 0xc00537a000)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x177›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0040c0000, {0xd1718f0, 0xc0079de840}, {0xc00378c540?}, 0xc00537a000)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/server/node.go:1374 +0x893›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0040c0000, {0xd1718f0, 0xc0079de3f0}, 0xc00537a000)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/server/node.go:1508 +0x465›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0xd1718f0, 0xc0079de3f0}, {0x980f220?, 0xc00537a000})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10423 +0x83›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000}, 0xc004be0d20, 0xc0217216e0)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x62c›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0x123›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000}, 0xc004be0d20, 0xc025e2b4c0)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0xf1›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0x123›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc0035b6000?, {{0x7f203075cf48?}, {0xd1ad1f0?, 0xc000c523c0?}}}, {0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000}, 0xc004be0d20, 0xc025e2b440)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x2f4›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0x123›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0xd1718f0, 0xc0079de3f0})›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x6b›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0033b8000, {0xd1718f0?, 0xc0079de3f0}, {0x7f20944f05b8?, 0x40?}, 0xc029b8f6b8)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0x186›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0xd1718f0, 0xc0079de3f0}, {0x980f220?, 0xc00537a000}, 0xc004be0d20, 0xc025e2b400)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x15c›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0xd1718f0, 0xc0079de3f0}, {0x980f220, 0xc00537a000}, 0x94a51a0?, 0x4e2fe9?)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0xd9›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x97d7e40?, 0xc0040c0000}, {0xd1718f0, 0xc0079de3f0}, 0xc012653ab0, 0xc0034012e0)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10425 +0x1c4›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.(*Server).processUnaryRPC(0xc003264780, {0xd1c2800, 0xc008a5e4e0}, 0xc0099f57a0, 0xc0017a0c00, 0x10f92e80, 0x0)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0x17d4›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.(*Server).handleStream(0xc003264780, {0xd1c2800, 0xc008a5e4e0}, 0xc0099f57a0, 0x0)›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0x1013›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹google.golang.org/grpc.(*Server).serveStreams.func1.2()›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0xfb›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 4707›
W231219 10:34:36.379831 219884 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [T1,Vsystem,n1,s1,r1/1:‹/{Min-System/NodeL…}›] 2439 +‹  google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x4a8›

I'll defer to @aadityasondhi and @nicktrav to whether this should block RC.

[yuzefovich]

I observed this in artifacts from #116761 (which is multi-tenant logic test run), similar to Renato's observation above - so perhaps the empty admission header problem is now limited only to multi-tenant deployments.

[aadityasondhi]

Thanks for pointing this out! This should NOT block RC. It is benign warning that only shows in tests: https://github.com/cockroachdb/cockroach/pull/115705.

Thanks for re-opening the issue though, I will investigate.

The default behaviour for AC with missing header is to skip AC altogether so no cause for alarm here.

[rafiss]

Seen in the artifacts of a demo test https://github.com/cockroachdb/cockroach/issues/117285#issue-2064450463:

W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 4980 [running]:›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹runtime/debug.Stack()›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    GOROOT/src/runtime/debug/stack.go:24 +0x64›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0x40017e8dc0, {0x723e420, 0x40066218f0}, {0x7209530, 0x4009f70150}, {0x1, {0x0, 0x0, 0x0}, {0x0, ...}, ...})›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1033 +0x284›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntents(...)›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:998›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntents(0x40063a4d58?, {0x723e420, 0x40066218f0}, {0x0, 0x0, 0x0, 0x0, 0x0}, {0x4009eb1580, 0x1, ...}, ...)›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:596 +0x2b0›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1.1({0x723e420?, 0x40066218f0?})›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:518 +0x64›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x723e3b0?, 0x400217f980?}, {0x5b83f1c, 0x17}, 0x6fc23ac00, 0x40063a4ec0)›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x90›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1({0x723e3b0, 0x400217f980})›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:516 +0xb8›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x128›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 4979›
W240103 18:08:07.207421 4980 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 705 +‹    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x330›

[srosenberg]

Saw this during a run of the kv0/enc=false/nodes=3/batch=16 roachtest (not a multi-tenant deployment),

W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 18941 [running]:›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹runtime/debug.Stack()›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   GOROOT/src/runtime/debug/stack.go:24 +0x64›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0x400166c780, {0x720d9a0, 0x4008221dc0}, {0x71d8db0, 0x40088636c8}, {0x1, {0x0, 0x0, 0x0}, {0x0, ...}, ...})›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1033 +0x284›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntents(...)›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:998›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntents(0x4002e81d58?, {0x720d9a0, 0x4008221dc0}, {0x0, 0x0, 0x0, 0x0, 0x0}, {0x400afb4900, 0x2, ...}, ...)›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:596 +0x2b0›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1.1({0x720d9a0?, 0x4008221dc0?})›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:518 +0x64›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x720d930?, 0x4003352360?}, {0x5b50f8a, 0x17}, 0x6fc23ac00, 0x4002e81ec0)›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x90›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1({0x720d930, 0x4003352360})›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:516 +0xb8›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x128›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18940›
W240109 07:25:38.868115 18941 kv/kvserver/intentresolver/intent_resolver.go:1032 ⋮ [-] 428 +‹   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x330›

[ajstorm]

What qualifies as test-only? We're seeing this on the DRT cluster, which was just upgraded to master

W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 73797719 [running]:›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹runtime/debug.Stack()›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     GOROOT/src/runtime/debug/stack.go:24 +0x5e›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc001837400, {0x78f0858, 0xc06c6334a0}, {0x78c07c8, 0xc0baaae410}, {0x1, {0x0, 0x0}, {0x0, 0x0, ...}, ...})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1029 +0x30d›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntent(_, {_, _}, {{{0xc01ea7bf20, 0xd, 0x10}, {0x0, 0x0, 0x0}}, {{0xff, ...}, ...}, ...}, ...)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:980 +0xf9›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc009aaf630, {0x78f0858, _}, {0xc0997d79e0, {0x17ab8e243cfaf175, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:661 +0x946›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn.func3({0x78f0890?, 0xc0d6a6b950?})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:401 +0x1cb›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc009aaf630, {0x78f0890, 0xc0d6a6b950}, {0xc0997d79e0, {0x17ab8e243cfaf175, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:447 +0x466›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc009aaf5e0, {0x78f0890, 0xc0d6a6b950}, 0xc0f6b46270, 0xc01d754b60?)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:350 +0x9ef›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc27a2f?, {0x78f0890, 0xc0d6a6b950}, 0xc0f6b46270?, {0xc0997d79e0, {0x17ab8e243cfaf175, 0x0}, 0x0, 0x0, 0x0, ...}, ...)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241 +0x2cc›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc009b4b900, {0x78f0890, 0xc0d6a6b950}, 0xc0997d78c0, 0x660cc90)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x32d›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc009b4b900, {0x78f0890?, 0xc0d6a6b920?}, 0xc0997d78c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x6c5›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc03bd4c000, {0x78f0890?, 0xc0d6a6b8f0?}, 0xc0997d78c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x6c5›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc03bd4c000, {0x78f0890?, 0xc0d6a6b8f0?}, 0xc0997d78c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x85c›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x3?, {0x78f0890, 0xc0d6a6b8f0}, 0xc0997d78c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x85›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc039af8c00, {0x78f0890?, 0xc0d6a6b890?}, {0x6?}, 0xc0997d78c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/server/node.go:1374 +0x545›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc039af8c00, {0x78f0890, 0xc0d6a6b740}, 0xc0997d78c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/server/node.go:1508 +0x2b5›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x78f0890, 0xc0d6a6b740}, {0x60afe80?, 0xc0997d78c0})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10427 +0x72›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x78f0890, 0xc0d6a6b740}, {0x60afe80, 0xc0997d78c0}, 0xc0386c8a00, 0xc0d61ee288)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b9›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x78f0890, 0xc0d6a6b740}, {0x60afe80, 0xc0997d78c0})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x78f0890, 0xc0d6a6b740}, {0x60afe80, 0xc0997d78c0}, 0xc0386c8a00?, 0xc0ac14b740)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x78f0890, 0xc0d6a6b740}, {0x60afe80, 0xc0997d78c0})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc003480000?, {{0x471aa5?}, {0x792ba68?, 0xc039c74630?}}}, {0x78f0890, 0xc0d6a6b5c0}, {0x60afe80, 0xc0997d78c0}, 0xc0386c8a00, 0xc0ac14b6c0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x1b3›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x78f0890, 0xc0d6a6b5c0}, {0x60afe80, 0xc0997d78c0})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x78f0890?, 0xc0d6a6b5c0?})›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001205710, {0x78f0890?, 0xc0d6a6b5c0}, {0xc064e93c00?, 0xc0278a9978?}, 0xc0278a9970)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x78f0890?, 0xc0d6a6b5c0?}, {0x60afe80?, 0xc0997d78c0?}, 0xc0386c8a00?, 0xc0d61ee288?)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x78f0890, 0xc0d6a6b5c0}, {0x60afe80, 0xc0997d78c0}, 0xc00c522a48?, 0x57c9d80?)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x85›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x60799a0?, 0xc039af8c00}, {0x78f0890, 0xc0d6a6b5c0}, 0xc048ac5810, 0xc038b01960)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10429 +0x135›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.(*Server).processUnaryRPC(0xc039aa12c0, {0x7940780, 0xc03b6e2b60}, 0xc07e00e900, 0xc03a4aaed0, 0xb336a20, 0x0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd67›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.(*Server).handleStream(0xc039aa12c0, {0x7940780, 0xc03b6e2b60}, 0xc07e00e900, 0x0)›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0x9e7›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹google.golang.org/grpc.(*Server).serveStreams.func1.2()›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x8d›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 279›
W240118 21:26:10.563153 73797719 kv/kvserver/intentresolver/intent_resolver.go:1028 ⋮ [T1,Vsystem,n10,tenant=3,s10,r26124/14:‹/Tenant/3/Table/208/1/115…›] 68980 +‹     google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x246›

[aadityasondhi]

What qualifies as test-only?

Any non-release build. So current master will show this message.

[dhartunian]

Just saw this log line in the output for this failure https://github.com/cockroachdb/cockroach/issues/120095

W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 21035 [running]:›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹runtime/debug.Stack()›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   GOROOT/src/runtime/debug/stack.go:24 +0x64›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0x40084aa640, {0x6068c90, 0x400676f5f0}, {0x603bdb0, 0x4006fa6c30}, {0x1, {0x0, 0x0}, {0x0, 0x0, ...}, ...})›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1039 +0x274›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntent(0x40084aa640, {0x6068c90, 0x400676f5f0}, {{{0x4005ecbe30, 0xe, 0x10}, {0x0, 0x0, 0x0}}, {{0x1, ...}, ...}, ...}, ...)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:990 +0xf0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0x40063b6e60, {0x6068c90, 0x400676f5f0}, {0x400632d7a0, {0x17bab5ceccdc40e0, 0x1}, 0x0, 0x0, 0x1, 0x0, ...}, ...)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:623 +0x6f0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0x40063b6e60, {0x6068c90, 0x400676f5f0}, {0x400632d7a0, {0x17bab5ceccdc40e0, 0x1}, 0x0, 0x0, 0x1, 0x0, ...}, ...)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:397 +0x3a8›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0x40063b6e10, {0x6068c90, 0x400676f5f0}, 0x400509a0d0, 0x3)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:353 +0x750›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0x40063b6e10, {0x6068c90, 0x400676f5f0}, 0x400509a0d0?, {0x400632d7a0, {0x17bab5ceccdc40e0, 0x1}, 0x0, 0x0, 0x1, ...}, ...)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x228›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0x4006049908, {0x6068c90, 0x400676f5f0}, 0x400632d680, 0x4f7cb80)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x1fc›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0x4006049908, {0x6068c90?, 0x400676f5c0?}, 0x400632d680)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x49c›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0x4004348008, {0x6068c90?, 0x400676f590?}, 0x400632d680)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x5b0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x4004370d00, {0x6068c90, 0x400676f590}, 0x400632d680)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0x60›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0x40049e9008, {0x6068c90?, 0x400676f560?}, {0x6?}, 0x400632d680)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/server/node.go:1388 +0x374›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0x40049e9008, {0x6068c90, 0x400676f4a0}, 0x400632d680)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/server/node.go:1522 +0x224›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x6068c90?, 0x400676f4a0?}, {0x4b8a340?, 0x400632d680?})›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10481 +0xd0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x6068c90, 0x400676f4a0}, {0x4b8a340, 0x400632d680}, 0x40042584c0, 0x40012b6fc0)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x32c›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x6068c90, 0x400676f4a0}, {0x4b8a340, 0x400632d680})›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xa0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x6068c90, 0x400676f4a0}, {0x4b8a340, 0x400632d680}, 0x40042584c0?, 0x4007c0d1c0)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x88›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x6068c90, 0x400676f4a0}, {0x4b8a340, 0x400632d680})›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xa0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0x4002538000?, {{0x48b7980?}, {0x60a4340?, 0x4007aa3860?}}}, {0x6068c90, 0x400676f380}, {0x4b8a340, 0x400632d680}, 0x40042584c0, 0x4007c0d180)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x15c›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x6068c90, 0x400676f380}, {0x4b8a340, 0x400632d680})›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xa0›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x6068c90?, 0x400676f380?})›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x40›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0x4003653200, {0x6068c90, 0x400676f380}, {0xb7c30c?, 0x38?}, 0x4006ddb960)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xac›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x6068c90?, 0x400676f380?}, {0x4b8a340?, 0x400632d680?}, 0x40042584c0?, 0x40012b6fc0?)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x7c›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x6068c90, 0x400676f380}, {0x4b8a340, 0x400632d680}, 0x40042584c0, 0x499f460?)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x88›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x4b6a120, 0x40049e9008}, {0x6068c90, 0x400676f380}, 0x400622ff10, 0x4003d10d80)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/aarch64-fastbuild/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10483 +0x148›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.(*Server).processUnaryRPC(0x400a1aa960, {0x60b5a20, 0x4008974680}, 0x400632d560, 0x400645a450, 0x8eff300, 0x0)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xa9c›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.(*Server).handleStream(0x400a1aa960, {0x60b5a20, 0x4008974680}, 0x400632d560, 0x0)›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0x7ac›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹google.golang.org/grpc.(*Server).serveStreams.func1.2()›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x84›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 20884›
W240308 06:21:48.371103 21035 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n1,tenant=10,s1,r68/1:‹/Tenant/1{0-1}›] 196 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x21c›

[rytaft]

Seen in https://github.com/cockroachdb/cockroach/issues/121176:

W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 18503 [running]:›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹runtime/debug.Stack()›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   GOROOT/src/runtime/debug/stack.go:24 +0x5e›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc001bfdcc0, {0x7faa128, 0xc001174070}, {0x7f77000, 0xc00c5e5ed8}, {0x1, {0x0, 0x0}, {0x0, 0x0, ...}, ...})›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1039 +0x311›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntents(...)›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1004›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntents(0xc001bfdcc0, {0x7faa128, 0xc001174070}, {0x0, 0x0, 0x0, 0x0, 0x0}, {0xc0093ff800, 0x2, ...}, ...)›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:602 +0x3e5›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1.1({0x7faa128?, 0xc001174070?})›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:524 +0x85›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x7faa0b8?, 0xc00b4b3d70?}, {0x676e6d2, 0x17}, 0x6fc23ac00, 0xc00e9ceec8)›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:29 +0x97›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1({0x7faa0b8, 0xc00b4b3d70})›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:522 +0xe9›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 18498›
W240327 06:06:03.499873 18503 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 656 +‹   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x3fe›

[srosenberg]

Seen in 121568:

W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 1454278774 [running]:›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹runtime/debug.Stack()›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      GOROOT/src/runtime/debug/stack.go:24 +0x5e›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc00132f2c0, {0x7ce9a98, 0xc0aa3e3810}, {0x7cb9220, 0xc0771d4a90}, {0x1, {0x0, 0x0}, {0x0, 0x0, ...}, ...})›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1039 +0x311›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntent(_, {_, _}, {{{0xc067fd97a0, 0xa, 0x10}, {0x0, 0x0, 0x0}}, {{0xeb, ...}, ...}, ...}, ...)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:990 +0xfc›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc065ec2d20, {0x7ce9a98, _}, {0xc02f3cb200, {0x17c27c168726ced6, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:623 +0x946›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn.func3({0x7ce9ad0?, 0xc05ed79260?})›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:363 +0x1cb›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc065ec2d20, {0x7ce9ad0, 0xc05ed79260}, {0xc02f3cb200, {0x17c27c168726ced6, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:409 +0x415›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc065ec2c80, {0x7ce9ad0, 0xc05ed79260}, 0xc0ce054680, 0x3)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:353 +0x9ec›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc065ec2c80, {0x7ce9ad0, 0xc05ed79260}, 0xc0ce054680?, {0xc02f3cb200, {0x17c27c168726ced6, 0x0}, 0x0, 0x0, 0x0, ...}, ...)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x2f2›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc010028d88, {0x7ce9ad0, 0xc05ed79260}, 0xc02f3cad80, 0x68d2d88)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x34c›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc010028d88, {0x7ce9ad0?, 0xc05ed79230?}, 0xc02f3cad80)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x685›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc003568a88, {0x7ce9ad0?, 0xc05ed79200?}, 0xc02f3cad80)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x83c›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc001745f00, {0x7ce9ad0, 0xc05ed79200}, 0xc02f3cad80)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0x7e›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc0037d2c08, {0x7ce9ad0?, 0xc05ed791d0?}, {0xc00382e780?}, 0xc02f3cad80)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/server/node.go:1431 +0x5b9›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc0037d2c08, {0x7ce9ad0, 0xc05ed79140}, 0xc02f3cad80)›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹      github.com/cockroachdb/cockroach/pkg/server/node.go:1625 +0x2da›
W240402 14:14:06.297361 1454278774 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n2,s2,r395/2:‹/Table/113/1/4{19/20…-23/44…}›] 50766 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x7ce9ad0?, 0xc05ed79140?}, {0x63fc300?, 0xc02f3cad80?})›

[ajstorm]

Is this issue something we're still tracking? We see these messages very frequently on the drt-large cluster. A random occurrence:

W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732  test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 106717155 [running]:›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹runtime/debug.Stack()›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   GOROOT/src/runtime/debug/stack.go:24 +0x5e›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0xc001b27680, {0x7cf1998, 0xc0eb8273b0}, {0x7cc1140, 0xc202dcf790}, {0x1, {0x0, 0x0}, {0x0, 0x0, ...}, ...})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1039 +0x311›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntent(_, {_, _}, {{{0xc184d0dad0, 0xe, 0x10}, {0x0, 0x0, 0x0}}, {{0xbc, ...}, ...}, ...}, ...)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:990 +0xfc›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).pushLockTxn(0xc03fd8f310, {0x7cf1998, _}, {0xc1619ee7e0, {0x17c5304a2e9c10cd, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:623 +0x946›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn.func3({0x7cf19d0?, 0xc1f4d36420?})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:363 +0x1cb›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableWaiterImpl).WaitOn(0xc03fd8f310, {0x7cf19d0, 0xc1f4d36420}, {0xc1619ee7e0, {0x17c5304a2e9c10cd, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table_waiter.go:409 +0x415›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc03fd8efa0, {0x7cf19d0, 0xc1f4d36420}, 0xc0ba056750, 0x3)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:353 +0x9ec›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0xc03fd8efa0, {0x7cf19d0, 0xc1f4d36420}, 0xc0ba056750?, {0xc1619ee7e0, {0x17c5304a2e9c10cd, 0x0}, 0x0, 0x0, 0x0, ...}, ...)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:244 +0x2f2›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc048bd6008, {0x7cf19d0, 0xc1f4d36420}, 0xc1619ee360, 0x68d5ca8)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x34c›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc048bd6008, {0x7cf19d0?, 0xc1f4d363f0?}, 0xc1619ee360)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:182 +0x685›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc00d3a2a88, {0x7cf19d0?, 0xc1f4d363c0?}, 0xc1619ee360)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x83c›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0xc00177b800, {0x7cf19d0, 0xc1f4d363c0}, 0xc1619ee360)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:203 +0x7e›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc003d00c08, {0x7cf19d0?, 0xc1f4d36390?}, {0xc003d6e8a0?}, 0xc1619ee360)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/server/node.go:1431 +0x5b9›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc003d00c08, {0x7cf19d0, 0xc1f4d36300}, 0xc1619ee360)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/server/node.go:1631 +0x2da›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler.func1({0x7cf19d0?, 0xc1f4d36300?}, {0x63feda0?, 0xc1619ee360?})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10584 +0xcb›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360}, 0xc1e4be7780, 0xc1f46302d0)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b2›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360}, 0xc1e4be7780?, 0xc1df882440)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc00219c000?, {{0x4707e5?}, {0x7d30c80?, 0xc003c75c20?}}}, {0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360}, 0xc1e4be7780, 0xc1df8821c0)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:95 +0x1e4›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.getChainUnaryHandler.func1({0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1163 +0xb2›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x7cf19d0?, 0xc1f4d36300?})›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc001c5e2d0, {0x7cf19d0, 0xc1f4d36300}, {0xc1df882000?, 0xc042fff978?}, 0xc042fff970)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:337 +0xcb›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x7cf19d0?, 0xc1f4d36300?}, {0x63feda0?, 0xc1619ee360?}, 0xc1e4be7780?, 0xc1f46302d0?)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x7cf19d0, 0xc1f4d36300}, {0x63feda0, 0xc1619ee360}, 0xc1e4be7780, 0x70?)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1154 +0x85›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹github.com/cockroachdb/cockroach/pkg/kv/kvpb._Internal_Batch_Handler({0x63cf6e0, 0xc003d00c08}, {0x7cf19d0, 0xc1f4d36300}, 0xc01c5667e0, 0xc003ce43e0)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   github.com/cockroachdb/cockroach/pkg/kv/kvpb/bazel-out/k8-opt/bin/pkg/kv/kvpb/kvpb_go_proto_/github.com/cockroachdb/cockroach/pkg/kv/kvpb/api.pb.go:10586 +0x143›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.(*Server).processUnaryRPC(0xc003cf23c0, {0x7d471a0, 0xc01198c820}, 0xc1619ee240, 0xc001822c60, 0xb8bc3e0, 0x0)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1336 +0xd16›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.(*Server).handleStream(0xc003cf23c0, {0x7d471a0, 0xc01198c820}, 0xc1619ee240, 0x0)›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:1704 +0x9da›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹google.golang.org/grpc.(*Server).serveStreams.func1.2()›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:965 +0x8d›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 5977›
W240411 09:38:48.560726 106717155 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [T1,Vsystem,n6,s89,r12983/8:‹/Table/134/1/"@"/12{37/…-55/…}›] 66732 +‹   google.golang.org/grpc/external/org_golang_google_grpc/server.go:963 +0x226›

262 occurrences in a single log file:

ubuntu@drt-large-0006:~/logs$ grep "test-only warning" cockroach.drt-large-0006.ubuntu.2024-04-11T09_38_29Z.009276.log | wc -l
262

[aadityasondhi]

It has fallen through the cracks. Mostly because we don't have a great way of finding the root of these missing headers, they come from separate goroutines. I found some here through code reading: https://github.com/cockroachdb/cockroach/pull/116286.

I can spend some time next week to dig deeper and see if I can find some more call sites that are missing these headers.

[ajstorm]

Thanks @aadityasondhi. For reference, we hit dozens of these every day on the DRT clusters. I'd suggest that you grep our logs if you're looking for more call sites.

[BabuSrithar]

Here is latest from drt-chaos cluster https://us5.datadoghq.com/logs?query=-status%3A%28info%20OR%20debug%29%20cluster%3Adrt-chaos&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&event=AgAAAZBnrQdZtckJMwAAAAAAAAAYAAAAAEFaQm5yUWRaQUFBZjM3dURqWFlfNlFCcQAAACQAAAAAMDE5MDY3YmEtOTUwMC00ZjI4LWI4YzctOGQxZTAzMjk3OTQ5&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=time%2Casc&viz=stream&from_ts=1719726300000&to_ts=1719727920000&live=false

[tbg]

Are we still collecting? Here's one from https://github.com/cockroachdb/cockroach/issues/129980

Details

``` W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 test-only warning: if you see this, please report to https://github.com/cockroachdb/cockroach/issues/112680. empty admission header provided by ‹goroutine 13191 [running]:› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹runtime/debug.Stack()› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ GOROOT/src/runtime/debug/stack.go:24 +0x64› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).resolveIntents(0x40019efe00, {0x7c9fda8, 0x400e88e850}, {0x7c66b18, 0x4016111548}, {0x1, {0x0, 0x 0}, {0x0, 0x0, ...}, ...})› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1039 +0x274› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).ResolveIntents(...)› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:1004› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntents(0x40019efe00, {0x7c9fda8, 0x400e88e850}, {0x0, 0x0, 0x0, 0x0, 0x0}, {0x400c2b6d00, 0x2, ...}, ...)› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:602 +0x2bc› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1.1({0x7c9fda8?, 0x400e88e850?})› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:524 +0x60› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/util/timeutil.RunWithTimeout({0x7c9fd70?, 0x401393c9f0?}, {0x637e82b, 0x17}, 0x6fc23ac00, 0x40034c5ec0)› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/util/timeutil/timeout.go:33 +0x80› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver.(*IntentResolver).CleanupIntentsAsync.func1({0x7c9fd70, 0x401393c9f0})› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/kv/kvserver/intentresolver/intent_resolver.go:522 +0xb0› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x128› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 13188› W240902 08:54:42.299319 13191 kv/kvserver/intentresolver/intent_resolver.go:1038 ⋮ [-] 214 +‹ github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x31c› ```