search

yarpc / yab

Call and benchmark YARPC services from the command line.
MIT License
84 stars 33 forks source link

Stalled gRPC server can result in reflection calls hanging (past specified timeout) #335

Closed prashantv closed 3 years ago

prashantv commented 3 years ago

Goroutine when yab is hung making a gRPC call to a "stuck" Java server,

goroutine 1 [select]:
google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc000040400)
        external/org_golang_google_grpc/internal/transport/transport.go:318 +0xd5
google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...)
        external/org_golang_google_grpc/internal/transport/transport.go:333
google.golang.org/grpc.(*csAttempt).recvMsg(0xc000160280, 0xe9aca0, 0xc000150320, 0x0, 0xc000038200, 0x5)
        external/org_golang_google_grpc/stream.go:870 +0x725
google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000160280, 0x203000, 0x203000)
        external/org_golang_google_grpc/stream.go:735 +0x46
google.golang.org/grpc.(*clientStream).withRetry(0xc0003bc7e0, 0xc0003a8bd8, 0xc0003a8ba8, 0xc000038200, 0x7fb3abd921a0)
        external/org_golang_google_grpc/stream.go:593 +0x9c
google.golang.org/grpc.(*clientStream).RecvMsg(0xc0003bc7e0, 0xe9aca0, 0xc000150320, 0xc0000e80c0, 0x0)
        external/org_golang_google_grpc/stream.go:734 +0x105
google.golang.org/grpc/reflection/grpc_reflection_v1alpha.(*serverReflectionServerReflectionInfoClient).Recv(0xc0000ac190, 0xc0000e80c0, 0x0, 0x0)
        external/org_golang_google_grpc/reflection/grpc_reflection_v1alpha/reflection.pb.go:686 +0x62
github.com/jhump/protoreflect/grpcreflect.(*Client).doSendLocked(0xc000160200, 0x1, 0xc0000e80c0, 0xc0003a8d28, 0x4149e5, 0xc0000f6418)
        external/com_github_jhump_protoreflect/grpcreflect/client.go:469 +0x1a2
github.com/jhump/protoreflect/grpcreflect.(*Client).doSend(0xc000160200, 0xea3801, 0xc0000e80c0, 0x0, 0x0, 0x0)
        external/com_github_jhump_protoreflect/grpcreflect/client.go:449 +0x90
github.com/jhump/protoreflect/grpcreflect.(*Client).send(0xc000160200, 0xc0000e80c0, 0x40, 0xe5a700, 0xc00000c040)
        external/com_github_jhump_protoreflect/grpcreflect/client.go:421 +0x3e
github.com/jhump/protoreflect/grpcreflect.(*Client).getAndCacheFileDescriptors(0xc000160200, 0xc0000e80c0, 0x0, 0x0, 0x0, 0x0, 0x7fb3d2a2c2a0, 0x18, 0x20)
        external/com_github_jhump_protoreflect/grpcreflect/client.go:239 +0x4d
github.com/jhump/protoreflect/grpcreflect.(*Client).FileContainingSymbol(0xc000160200, 0x7fff76cd57c4, 0x15, 0x44d66c, 0x20, 0xd5e000)
        external/com_github_jhump_protoreflect/grpcreflect/client.go:200 +0x13f
github.com/jhump/protoreflect/grpcreflect.(*Client).ResolveService(0xc000160200, 0x7fff76cd57c4, 0x15, 0x15, 0xee2ba5, 0x1)
        external/com_github_jhump_protoreflect/grpcreflect/client.go:519 +0x45
github.com/yarpc/yab/protobuf.(*grpcreflectSource).FindService(0xc00000e020, 0x7fff76cd57c4, 0x15, 0x15, 0x7fff76cd57da, 0x5)
        external/com_github_yarpc_yab/protobuf/source_reflection.go:90 +0x5d
github.com/yarpc/yab/encoding.NewProtobuf(0x7fff76cd57c4, 0x1b, 0xffe760, 0xc00000e020, 0x0, 0x0, 0x0, 0x7fff76cd57c4)
        external/com_github_yarpc_yab/encoding/protobuf.go:85 +0xac
main.NewSerializer(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7fff76cd57c4, 0x1b, 0xc000496038, ...)
        external/com_github_yarpc_yab/request.go:128 +0x3ae
main.runWithOptions(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7fff76cd57c4, 0x1b, 0xc000496038, ...)
        external/com_github_yarpc_yab/main.go:327 +0x4f0
main.parseAndRun(0x1003860, 0xc000140008)
        external/com_github_yarpc_yab/main.go:218 +0x4a8
main.main()
        external/com_github_yarpc_yab/main.go:85 +0x52

The reflection call is hung, it doesn't seem to respect the timeout.

(Related to internal task, FPPF-936)

rabbbit commented 3 years ago

Was planning to raise this, forgot - thx!