search

grpc-ecosystem / grpc-spring

Spring Boot starter module for gRPC framework.
https://grpc-ecosystem.github.io/grpc-spring/
Apache License 2.0
3.52k stars 826 forks source link

grpc server discovery #426

Closed johnyq closed 4 years ago

johnyq commented 4 years ago

I have a very strange problem when use the nacos as discovery server in the k8s environment.

  1. all the service running in the k8s .client call the service , like this @GrpcClient("resource-service") and the application.properties config grpc.client.resource-service.address=dns:///resources-service:9090 "resources-service" is k8s service name . this is ok.

  2. use nacos as service discovery server. remove grpc client target server property.All the services running, nacos dashboard display its at normal state,local test is working.

  3. Deploy to the K8S ,call the resource-service failed. I check code and configuration and try more times and get the same result. when i recover dns configuration property, the code working. following logs

2020-09-09 10:13:02,924 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] OUTBOUND HEADERS: streamId=5 headers=GrpcHttp2OutboundHeaders[:authority: resource-service, :path: /SystemConfigService/getAllConfigSet, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.31.1, user-bin: CiA4YWFhMTUzNzczZmI4MDc2MDE3M2ZiODM2NmQxMDAwNhIHcWlhbmhhbxoG6ZKx5rWpIg5saWhAZ2lzdW5pLmNvbSgsMgzpkrHmtannu4Tnu4c6IGRmMGJhOTM3Yzk1YjQ2NGI5OGZmNWNhYzNiMTRlZTBjQg/ns7vnu5/nrqHnkIblkZhIAlIGZEFwdCJ4, grpc-accept-encoding: gzip] streamDependency=0 weight=16 exclusive=false padding=0 endStream=false 2020-09-09 10:13:02,925 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] OUTBOUND DATA: streamId=5 padding=0 endStream=true length=5 bytes=0000000000 2020-09-09 10:13:02,933 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] INBOUND HEADERS: streamId=5 headers=GrpcHttp2ResponseHeaders[:status: 200, content-type: application/grpc, grpc-encoding: identity, grpc-accept-encoding: gzip, x-envoy-upstream-service-time: 7, date: Wed, 09 Sep 2020 02:13:02 GMT, server: envoy] padding=0 endStream=false 2020-09-09 10:13:02,934 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] INBOUND DATA: streamId=5 padding=0 endStream=false length=3711 bytes=0000000e7a0a3d1209e4baa7e59381e5908d1a1073797374656d5f696e666f5f6e616d6522134749537461636b20666f72204d616e616765722a09e4baa7e593... 2020-09-09 10:13:02,934 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] OUTBOUND PING: ack=false bytes=1234 2020-09-09 10:13:02,934 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] INBOUND DATA: streamId=5 padding=0 endStream=true length=0 bytes= 2020-09-09 10:13:02,934 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] INBOUND PING: ack=true bytes=1234 2020-09-09 10:13:02,934 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - Window: 1048576 2020-09-09 10:13:02,935 [grpc-default-worker-ELG-1-1] DEBUG i.g.n.s.i.g.n.NettyClientHandler - [id: 0x5fb03800, L:/10.42.172.141:34938 - R:10.42.172.190/10.42.172.190:9090] OUTBOUND RST_STREAM: streamId=5 errorCode=8 2020-09-09 10:13:02,936 [XNIO-2 task-4] ERROR c.g.g.c.h.GistackGlobalExceptionHandler - grpcServer runtime exception : io.grpc.StatusRuntimeException: INTERNAL: Received unexpected EOS on DATA frame from server. at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)

errorCode=8

At first I thought it was network error or grpc-java issue until I back to "grpc.client.....= dns:///" . I'm not sure what it is what's causing it

The application's environment spring boot 2.1.16.RELEASE spring cloud Greenwich.SR6 spring cloud alibaba 2.1.2.RELEASE grpc-spring-boot-start 2.10.1.RELEASE

ST-DDT commented 4 years ago

IIRC there is a slight difference between the dns and the discovery resolver. The first one connects to the k8s-service whereas the second one connects to the k8s-pods/the ips provided by the discovery service. Have you tried "downloading" the resources-service ips from your discovery and connect to them directly either using: static://<ip:port>,... (supports multiple) or dns:///<ip:port> (supports only one)? Then we can check whether it's an issue with the connection/network or the discovery-service/resolver.

ST-DDT commented 4 years ago

Closed until required information are provided