HTTP/2 client preface string missing or corrupt

[xiaoqian2006]

The context

The client side is on kylin10-arm-34 to connect to other nodes. Among them, tdh-01~ tdh-02 are the same operating system, but tdh-01 cannot connect.

The bug

Stacktrace and logs

server[tdh-01] error:

2024-02-26 13:42:00,494 INFO [main] n.d.b.g.s.s.GrpcServerLifecycle: gRPC Server started, listening on address: *, port: 10208
2024-02-26 13:42:00,511 INFO [main] io.transwarp.manager.agent.ManagerAgent: Started ManagerAgent in 3.585 seconds (JVM running for 4.463)
2024-02-26 13:42:01,430 INFO [grpc-default-worker-ELG-3-1] i.g.n.s.i.g.n.N.connections: Transport failed
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception: HTTP/2 client preface string missing or corrupt. Hex dump for received bytes: 0000002f122d080112290a1e0a076d616e61676572120b31
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.readClientPrefaceString(Http2ConnectionHandler.java:321)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:247)
    at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
    at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
    at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
    at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
    at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
    at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
    at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:829)

client error:

2024-02-26 13:41:33,826 ERROR [http-nio-8180-exec-4] i.t.manager.master.remote.AgentService: Fail to get hardware info at node(tdh-01).
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [closed=[CANCELLED], committed=[remote_addr=tdh-01/172.16.251.240:10208]]
        at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268)
        at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167)
        at io.transwarp.manager.common.protocol.sys.SysServiceGrpc$SysServiceBlockingStub.getSysInfo(SysServiceGrpc.java:782)
        at io.transwarp.manager.master.remote.AgentService.getSysInfo(AgentService.java:293)
        at io.transwarp.manager.master.node.NodeService$SysInfoHelper.loadSysInfo(NodeService.java:628)
        at io.transwarp.manager.master.node.NodeService$SysInfoHelper.getSysInfo(NodeService.java:605)
        at io.transwarp.manager.master.node.NodeService$SysInfoHelper$$FastClassBySpringCGLIB$$375e7171.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.cache.interceptor.CacheInterceptor.lambda$invoke$0(CacheInterceptor.java:54)
        at org.springframework.cache.interceptor.CacheAspectSupport.invokeOperation(CacheAspectSupport.java:366)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:420)
        at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:345)
        at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:64)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707)
        at io.transwarp.manager.master.node.NodeService$SysInfoHelper$$EnhancerBySpringCGLIB$$b77112ce.getSysInfo(<generated>)
        at io.transwarp.manager.master.node.NodeService.getSysInfo(NodeService.java:505)
        at io.transwarp.manager.master.node.NodeService.toNodeInfos(NodeService.java:232)
        at io.transwarp.manager.master.node.NodeService$$FastClassBySpringCGLIB$$d6ce278c.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386)
        at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703)
        at io.transwarp.manager.master.node.NodeService$$EnhancerBySpringCGLIB$$a8773421.toNodeInfos(<generated>)
        at io.transwarp.manager.master.node.NodeController.getPagedNodes(NodeController.java:479)
        at io.transwarp.manager.master.node.NodeController$$FastClassBySpringCGLIB$$ad39bf65.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792)
        at org.springframework.aop.framework.ReflectiveMeth

Steps to Reproduce

client code:

server code:

The application's environment

1. java11
2. kylin10-arm-34：Kylin Linux Advanced Server V10 (Tercel) arm
3. tdh-01 ～ tdh-03： CentOS Linux 7.6 1810 x86_64 (Only modified/etc/redhat-release, disguised as CentOS release 6.5 (Final))

Which versions do you use?

1. 
2. 

Additional context

server application.properties:

client application.properties:

none（Connect to the server via hostname and port.）

Did it ever work before?

Do you have a demo?

[ST-DDT]

If you look at the hex dump: 0000002f122d080112290a1e0a076d616e61676572120b31 And try to decode it you get some binary garbage. Is it possible that the client tries to connect to the server using TLS encryption but the server is in plaintext mode? Please check the negotiationType property for more information. https://grpc-ecosystem.github.io/grpc-spring/en/client/security.html

[xiaoqian2006]

If you look at the hex dump: 0000002f122d080112290a1e0a076d616e61676572120b31 And try to decode it you get some binary garbage. Is it possible that the client tries to connect to the server using TLS encryption but the server is in plaintext mode? Please check the negotiationType property for more information. https://grpc-ecosystem.github.io/grpc-spring/en/client/security.html

If I want to configure client Disable Transport Layer SecurityDisable Transport Layer Security through “grpc.client.name.negotiationType=PLAINTEXT”, what should I “name” in my client and server connection method via hostname and port?

[xiaoqian2006]

我已经在client side设置了grpc.client.xxx.negotiationType=PLAINTEXT但是依然有Http2Exception · Issue #648 · grpc-ecosystem/grpc-spring (github.com)

I have tried setting the client according to this issue, but it doesn't seem to work.

[xiaoqian2006]

And on the client side, usePlaintext has been executed when creating the Channel.

[xiaoqian2006]

It is very strange that tdh-01~ tdh03 are both the same operating system and the same deployment method. Why is only tdh-01 connection reporting an error?