search

pinpoint-apm / pinpoint-docker

Official Dockerized components of the Pinpoint
http://pinpoint-apm.github.io/pinpoint/
Apache License 2.0
461 stars 227 forks source link

2020-03-24 15:14:20 [INFO ](c.n.p.p.s.TcpDataSender@Default ) request fail. request:com.navercorp.pinpoint.profiler.metadata.ApiMetaData@e3c30ea Caused:timeout com.navercorp.pinpoint.rpc.PinpointSocketException: timeout #56

Open Jcmouy opened 4 years ago

Jcmouy commented 4 years ago

Hello, thank you very much for having solved the previous problems. I want to consult a particular situation that is happening to me, I downloaded the latest version of pinpoint docker, and everything is working correctly including the test app, but when I want to link the application I have, the server throws the following message:

2020-03-24 15:14:20 [INFO ](c.n.p.p.s.TcpDataSender@Default    ) request fail. request:com.navercorp.pinpoint.profiler.metadata.ApiMetaData@e3c30ea Caused:timeout
com.navercorp.pinpoint.rpc.PinpointSocketException: timeout
    at com.navercorp.pinpoint.rpc.DefaultFuture.fireTimeout(DefaultFuture.java:120)
    at com.navercorp.pinpoint.rpc.DefaultFuture.run(DefaultFuture.java:235)
    at org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:556)
    at org.jboss.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:632)
    at org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:369)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at java.lang.Thread.run(Thread.java:745)

I tested networktest.sh and the result was successful, I'm currently using the THRIFT protocol. I would be grateful if someone could guide me towards solving this problem.

Thank you very much.

RoySRose commented 4 years ago

Hello, @Jcmouy

I can't only assume this is about network problem. As you've mentioned that everything is working with the test app. I think you need to check the fire wall of any other containers you've started.

And why don't you use GRPC as default. Since Pinpoint uses GRPC as default from v2

Jcmouy commented 4 years ago

Hi, Roy, thank you so much for answering. The networking area does not fall under my section, I still consulted the person in charge and was informed that there is no firewall between the computers and the ports are enabled, so that should not be the problem. In any case I ran the network test and it gave the following result:

`[e990159@vmlx-jbossrh-desa script]$ sh networktest.sh 
CLASSPATH=./tools/pinpoint-tools-2.0.1.jar:
2020-03-24 15:31:26 [INFO ](com.navercorp.pinpoint.bootstrap.config.DefaultProfilerConfig) configuration loaded successfully.
UDP-STAT:// 10.97.244.126
    => 10.97.244.126:9995 [SUCCESS]

UDP-SPAN:// 10.97.244.126
    => 10.97.244.126:9996 [SUCCESS]

TCP:// 10.97.244.126
    => 10.97.244.126:9994 [SUCCESS]
`

Which gives me to understand that at that point is not the problem, I copy the logs of the application in the server, the collector and the agent in case they are providing information that I'm not seeing.

Application server:

`2020-03-26 22:37:27 [INFO ](c.n.p.g.c.DefaultChannelFactory    ) Set nameResolverProvider io.grpc.internal.PinpointDnsNameResolverProvider@287f94b1. channelName=AgentGrpcDataSender, host=10.97.244.126, port=9991
2020-03-26 22:37:27 [INFO ](c.n.p.g.c.DefaultChannelFactory    ) Set clientOption ClientOption{keepAliveTime=30000, keepAliveTimeout=60000, keepAliveWithoutCalls=false, idleTimeoutMillis=2592000000, maxHeaderListSize=8192, maxInboundMessageSize=4194304, flowControlWindow=1048576, connectTimeout=20000, writeBufferHighWaterMark=33554432, writeBufferLowWaterMark=16777216}. name=AgentGrpcDataSender
2020-03-26 22:37:27 [INFO ](c.n.p.p.s.g.ReconnectExecutor      ) newReconnector(reconnectJob = [com.navercorp.pinpoint.profiler.receiver.grpc.GrpcCommandService$1@43d455c9])
2020-03-26 22:37:27 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.ReconnectExecutor      ) newReconnector(reconnectJob = [com.navercorp.pinpoint.profiler.sender.grpc.AgentGrpcDataSender$1@4aeaadc1])
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.AgentGrpcDataSender    ) newPingStream:PingStreamContext{PingStream-2}
2020-03-26 22:37:28 [INFO ](.p.g.c.DefaultChannelFactoryBuilder) build ChannelFactory:StatGrpcDataSender
2020-03-26 22:37:28 [INFO ](c.n.p.g.c.DefaultChannelFactory    ) Set nameResolverProvider io.grpc.internal.PinpointDnsNameResolverProvider@287f94b1. channelName=StatGrpcDataSender, host=10.97.244.126, port=9992
2020-03-26 22:37:28 [INFO ](c.n.p.g.c.DefaultChannelFactory    ) Set clientOption ClientOption{keepAliveTime=30000, keepAliveTimeout=60000, keepAliveWithoutCalls=false, idleTimeoutMillis=2592000000, maxHeaderListSize=8192, maxInboundMessageSize=4194304, flowControlWindow=1048576, connectTimeout=20000, writeBufferHighWaterMark=33554432, writeBufferLowWaterMark=16777216}. name=StatGrpcDataSender
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.ReconnectExecutor      ) newReconnector(reconnectJob = [com.navercorp.pinpoint.profiler.sender.grpc.StatGrpcDataSender$1@32fdec40])
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:StatStream-3
2020-03-26 22:37:28 [INFO ](.n.p.p.c.p.s.j.MemoryMetricProvider) loaded : Default memory metric
2020-03-26 22:37:28 [INFO ](n.p.p.c.p.s.c.CpuLoadMetricProvider) loaded : CpuLoadMetric for Oracle Java 1.7+
2020-03-26 22:37:28 [INFO ](.p.c.p.s.d.DataSourceMetricProvider) loaded : Default DataSourceMetric
2020-03-26 22:37:28 [INFO ](.p.s.f.FileDescriptorMetricProvider) loaded : FileDescriptorMetric for Oracle Java 1.5+
2020-03-26 22:37:28 [INFO ](.n.p.p.c.p.s.b.BufferMetricProvider) loaded : DefaultBufferMetric
2020-03-26 22:37:28 [INFO ](.p.s.j.DetailedMemoryMetricProvider) loaded : CMS detailed memory metric
2020-03-26 22:37:28 [INFO ](ailedGarbageCollectorMetricProvider) loaded : CMS detailed garbage collector metric
2020-03-26 22:37:28 [INFO ](n.p.p.c.m.DefaultApplicationContext) rpcModuleLifeCycle:GrpcModuleLifeCycle{, agentDataSender=null, metadataDataSender=null, spanDataSender=null, statDataSender=null, dnsExecutorService=null, reconnectScheduledExecutorService=null}
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) start()
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) agetInfoDataSenderProvider:com.navercorp.pinpoint.profiler.sender.grpc.AgentGrpcDataSender@49096b06
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) metadataDataSenderProvider:com.navercorp.pinpoint.profiler.sender.grpc.MetadataGrpcDataSender@4a183d02
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) spanDataSenderProvider:SpanGrpcDataSender{name='SpanGrpcDataSender', host='10.97.244.126', port=9993} com.navercorp.pinpoint.profiler.sender.grpc.SpanGrpcDataSender@5d05ef57
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) statDataSenderProvider:StatGrpcDataSender{name='StatGrpcDataSender', host='10.97.244.126', port=9992} com.navercorp.pinpoint.profiler.sender.grpc.StatGrpcDataSender@213deac2
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) dnsExecutorServiceProvider:java.util.concurrent.ThreadPoolExecutor@23eee4b8[Running, pool size = 4, active threads = 4, queued tasks = 0, completed tasks = 0]
2020-03-26 22:37:28 [INFO ](c.n.p.p.c.m.GrpcModuleLifeCycle    ) reconnectScheduledExecutorServiceProvider:java.util.concurrent.Executors$DelegatedScheduledExecutorService@28952dea
2020-03-26 22:37:28 [INFO ](n.p.p.c.m.DefaultApplicationContext) agentInformation:DefaultAgentInformation{agentId='web-user-desa-n1', applicationName='web-user-desa', isContainer=false, startTime=1585273042156, pid=9225, machineName='vmlx-jbossrh-desa', hostIp='10.97.195.13', serverType=JBOSS, jvmVersion='1.8.0_91', agentVersion='2.0.1'}
2020-03-26 22:37:28 [INFO ](c.n.p.p.DefaultAgent               ) Starting pinpoint Agent.
2020-03-26 22:37:28 [INFO ](.r.DefaultInterceptorRegistryBinder) bind:DefaultRegistry-0
2020-03-26 22:37:28 [INFO ](c.n.p.p.m.DefaultDeadlockMonitor   ) DefaultDeadlockMonitor started
2020-03-26 22:37:28 [INFO ](c.n.p.p.m.DefaultAgentStatMonitor  ) AgentStat monitor started
2020-03-26 22:37:28 [INFO ](c.n.p.p.DefaultAgent               ) registerStopHandler
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.Java7ShutdownHookRegister) register() started.
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.Java7ShutdownHookRegister) register() completed.
2020-03-26 22:37:28 [INFO ](c.n.p.p.AgentInfoSender            ) Sending AgentInfo AgentInfo{agentInformation=DefaultAgentInformation{agentId='web-user-desa-n1', applicationName='web-user-desa', isContainer=false, startTime=1585273042156, pid=9225, machineName='vmlx-jbossrh-desa', hostIp='10.97.195.13', serverType=JBOSS, jvmVersion='1.8.0_91', agentVersion='2.0.1'}, serverMetaData=DefaultServerMetaData{serverInfo='', vmArgs=[-D[Standalone], -Xrs, -Xms1024m, -Xmx2048m, -XX:MetaspaceSize=512m, -XX:MaxMetaspaceSize=512m, -XX:+PrintCompilation, -XX:InitialCodeCacheSize=128m, -XX:ReservedCodeCacheSize=256m, -XX:+UseCodeCacheFlushing, -XX:NewSize=682m, -XX:MaxNewSize=682m, -XX:SurvivorRatio=20, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, -XX:+CMSParallelRemarkEnabled, -Djboss.socket.binding.port-offset=13000, -Djboss.partition.name=DUTIA-cluster, -Djboss.partition.udpGroup=230.10.11.102, -Djboss.default.multicast.address=230.10.11.102, -Djboss.messaging.group.address=230.10.11.102, -Djboss.messaging.cluster.password=dwmmpass, -Djavax.net.ssl.trustStore=/opt/apps/keystores/cacerts, -Djavax.net.ssl.trustStorePassword=changeit, -Dfile.encoding=ISO-8859-1, -Dsun.jnu.encoding=ISO-8859-1, -Djava.net.preferIPv4Stack=true, -Dorg.jboss.resolver.warning=true, -Dsun.rmi.dgc.client.gcInterval=3600000, -Dsun.rmi.dgc.server.gcInterval=3600000, -Djboss.modules.system.pkgs=org.jboss.byteman, -Djava.awt.headless=true, -Djboss.server.default.config=standalone-full-ha.xml, -Duser.timezone=America/Montevideo, -Duser.language=en, -Duser.country=UY, -Djboss.server.log.threshold=INFO, -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager,com.navercorp.pinpoint.bootstrap,com.navercorp.pinpoint.common,com.navercorp.pinpoint.exception, -Djava.util.logging.manager=org.jboss.logmanager.LogManager, -Xbootclasspath/p:/opt/apps/wildfly-10.1.0.Final/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-2.0.4.Final.jar, -javaagent:/opt/apps/Applications10/DUTIAn1-server/WorkDir/pinpoint/pinpoint-bootstrap-1.9.0-SNAPSHOT.jar, -Dpinpoint.applicationName=web-user-desa, -Dpinpoint.agentId=web-user-desa-n1, -Djboss.server.base.dir=/opt/apps/wildfly-10.1.0.Final/DUTIAn1-server, -Djboss.server.log.dir=/opt/apps/Applications10/DUTIAn1-server/Logs, -Djboss.node.name=DUTIAn1-server, -Djboss.messaging.ServerPeerID=0, -DjvmRoute=DUTIAn1-serverajp13, -XX:ParallelGCThreads=4, -Dorg.jboss.boot.log.file=/opt/apps/Applications10/DUTIAn1-server/Logs/server.log, -Dlogging.configuration=file:/opt/apps/wildfly-10.1.0.Final/DUTIAn1-server/configuration/logging.properties], connectors={}, serviceInfo=[]}, jvmInformation=com.navercorp.pinpoint.profiler.JvmInformation@43947d41}
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) onReadyHandler:SpanStream-1 eventNumber:0
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) onReadyHandler:StatStream-3 eventNumber:0
2020-03-26 22:37:28 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Connect to CommandServiceStream completed.
2020-03-26 22:37:28 [INFO ](c.n.p.p.s.g.PingStreamContext      ) PingStream-2 onReady
2020-03-26 22:37:31 [WARN ](c.n.p.p.AgentInfoSender            ) request timed out while waiting for response.
2020-03-26 22:37:39 [INFO ](c.n.p.p.i.ASMEngine                ) appendToBootstrapClassPath:/opt/apps/Applications10/DUTIAn1-server/WorkDir/pinpoint/plugin/pinpoint-jdk-completable-future-plugin-2.0.1.jar
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 27 apiInfo: "com.google.gson.Gson.fromJson(java.lang.String json, java.lang.reflect.Type typeOfT)" line: 797, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999864801ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999864801ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 28 apiInfo: "com.google.gson.Gson.fromJson(java.io.Reader json, java.lang.Class classOfT)" line: 824, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999855598ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999855598ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 29 apiInfo: "com.google.gson.Gson.fromJson(java.io.Reader json, java.lang.reflect.Type typeOfT)" line: 851, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999868323ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999868323ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 30 apiInfo: "com.google.gson.Gson.fromJson(com.google.gson.stream.JsonReader reader, java.lang.reflect.Type typeOfT)" line: 879, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999878118ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999878118ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 31 apiInfo: "com.google.gson.Gson.fromJson(com.google.gson.JsonElement json, java.lang.Class classOfT)" line: 925, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999848795ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999848795ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 32 apiInfo: "com.google.gson.Gson.fromJson(com.google.gson.JsonElement json, java.lang.reflect.Type typeOfT)" line: 949, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999858258ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999858258ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 33 apiInfo: "com.google.gson.Gson.toJson(java.lang.Object src)" line: 580, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999811328ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999811328ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 34 apiInfo: "com.google.gson.Gson.toJson(java.lang.Object src, java.lang.reflect.Type typeOfSrc)" line: 602, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999866124ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999866124ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 35 apiInfo: "com.google.gson.Gson.toJson(java.lang.Object src, java.lang.Appendable writer)" line: 622, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999855566ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999855566ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 36 apiInfo: "com.google.gson.Gson.toJson(java.lang.Object src, java.lang.reflect.Type typeOfSrc, java.lang.Appendable writer)" line: 647, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999836344ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999836344ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 37 apiInfo: "com.google.gson.Gson.toJson(java.lang.Object src, java.lang.reflect.Type typeOfSrc, com.google.gson.stream.JsonWriter writer)" line: 661, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999853040ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999853040ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 38 apiInfo: "com.google.gson.Gson.toJson(com.google.gson.JsonElement jsonElement)" line: 687, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999876925ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999876925ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 39 apiInfo: "com.google.gson.Gson.toJson(com.google.gson.JsonElement jsonElement, java.lang.Appendable writer)" line: 702, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999812469ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999812469ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 40 apiInfo: "com.google.gson.Gson.toJson(com.google.gson.JsonElement jsonElement, com.google.gson.stream.JsonWriter writer)" line: 738, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999870959ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999870959ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 41 apiInfo: "org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(org.apache.cxf.message.Message message)" line: 50, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999676713ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999676713ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:38:11 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 42 apiInfo: "org.apache.cxf.interceptor.MessageSenderInterceptor.handleMessage(org.apache.cxf.message.Message message)" line: 46, cause=DEADLINE_EXCEEDED: deadline exceeded after 14999672194ns
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 14999672194ns
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:39:28 [WARN ](c.n.p.p.r.g.GrpcCommandService     ) Failed to command stream, cause=UNAVAILABLE: Keepalive failed. The connection is likely gone
io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:39:28 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() started
2020-03-26 22:39:28 [INFO ](.p.r.g.GrpcActiveThreadCountService) close
2020-03-26 22:39:28 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() completed
2020-03-26 22:39:28 [WARN ](c.n.p.p.s.g.PingStreamContext      ) Failed to ping stream, streamId=PingStream-2, cause=UNAVAILABLE: Keepalive failed. The connection is likely gone
io.grpc.StatusRuntimeException: UNAVAILABLE: Keepalive failed. The connection is likely gone
    at io.grpc.Status.asRuntimeException(Status.java:526)
    at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
    at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
    at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
    at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
    at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
    at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2020-03-26 22:39:29 [INFO ](c.n.p.p.s.g.AgentGrpcDataSender    ) newPingStream:PingStreamContext{PingStream-4}
2020-03-26 22:39:29 [INFO ](c.n.p.p.s.g.PingStreamContext      ) PingStream-4 onReady
2020-03-26 22:39:29 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-26 22:39:29 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Connect to CommandServiceStream completed.
2020-03-26 22:39:29 [INFO ](c.n.p.p.s.g.PingStreamContext      ) PingStream-4 success:`

Collector logs:

`2020-03-27 01:31:58 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/172.24.0.12:33146}
2020-03-27 01:31:58 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/172.24.0.12:53340}
2020-03-27 01:31:58 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/172.24.0.12:35650}
2020-03-27 01:32:02 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/172.24.0.12:33152}
2020-03-27 01:32:28 [DEBUG](c.n.p.c.r.g.s.c.GrpcCommandService ) AgentInfo{quickapp:app-in-docker:1585272660934:} => local. handleCommand(). transportId:3
2020-03-27 01:32:28 [INFO ](c.n.p.c.r.g.s.c.GrpcCommandService ) quickapp:app-in-docker:1585272660934 => local. ready() transportId:3
2020-03-27 01:32:28 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:CONNECTED ,before:NONE ,current:CONNECTED).
2020-03-27 01:32:28 [INFO ](c.n.p.c.r.g.s.c.GrpcCommandService ) quickapp:app-in-docker:1585272660934 => local. execute handshake:[740, 710, 730, 750]
2020-03-27 01:32:28 [INFO ](c.n.p.c.r.g.PinpointGrpcServer     ) AgentInfo{quickapp:app-in-docker:1585272660934:} handleHandshake() started. data:[740, 710, 730, 750]
2020-03-27 01:32:28 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:RUN_WITHOUT_HANDSHAKE ,before:CONNECTED ,current:RUN_WITHOUT_HANDSHAKE).
2020-03-27 01:32:28 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:RUN_DUPLEX ,before:RUN_WITHOUT_HANDSHAKE ,current:RUN_DUPLEX).
2020-03-27 01:32:28 [DEBUG](c.n.p.c.c.z.ZookeeperJobWorker     ) addPinpointServer key:quickapp:app-in-docker:1585272660934
2020-03-27 01:32:28 [DEBUG](c.n.p.c.c.z.ZookeeperJobWorker     ) handleUpdate zookeeperJobList:[ZookeeperJob{type=ADD, key='quickapp:app-in-docker:1585272660934'}]
2020-03-27 01:32:43 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272737920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73700688 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57670104 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.793969849246232E-4 systemCpuLoad: 0.022738693467336684 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272742920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73700848 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57741912 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.00401455275373228 systemCpuLoad: 0.02746425884123401 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272747920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701008 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57762784 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001006036217303823 systemCpuLoad: 0.028808655176751796 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272752920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701168 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57847928 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.002012072434607646 systemCpuLoad: 0.04904426559356137 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272757920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701328 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57847928 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.017561465127947E-4 systemCpuLoad: 0.02521324636226794 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272762921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701488 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57874288 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.527286413248024E-4 systemCpuLoad: 0.0191921726041144 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:32:43 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:33:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272767920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701648 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57457392 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0013810420590081606 systemCpuLoad: 0.041436464088397795 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272772920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701808 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57457392 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.7716872014080966E-4 systemCpuLoad: 0.028284098051539912 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272777920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73701968 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57458160 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.28140703517588E-4 systemCpuLoad: 0.025631360723709008 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272782920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73702128 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57471992 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.776328986960882E-4 systemCpuLoad: 0.022063432368058168 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272787920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73702288 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57479240 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.796179944709725E-4 systemCpuLoad: 0.02727158476812869 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272792920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73702448 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57481288 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.532956685499059E-4 systemCpuLoad: 0.020213433772755807 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:33:13 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:33:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272797920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73702608 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57497320 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0012550200803212851 systemCpuLoad: 0.02484004516371848 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272802920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73702768 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57498024 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.011903270266884E-4 systemCpuLoad: 0.020052638175209928 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272807920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73702928 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57507352 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010050251256281408 systemCpuLoad: 0.022741550446035937 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272812920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 73703088 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57510104 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010031347962382445 systemCpuLoad: 0.020561685055165497 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272817921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57526496 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.530120481927711E-4 systemCpuLoad: 0.01869979919678715 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272822920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57545288 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271165182490907E-4 systemCpuLoad: 0.019192172604114398 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:33:43 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:34:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272827921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57549504 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.520681875156681E-4 systemCpuLoad: 0.02469290549009777 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272832920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57551704 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.02071043052592E-4 systemCpuLoad: 0.025354587674155896 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272837920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57551704 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.02071043052592E-4 systemCpuLoad: 0.017193775100401605 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272842920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57583328 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001377928097206564 systemCpuLoad: 0.02179904785767978 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272847920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57583328 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.017561465127947E-4 systemCpuLoad: 0.019189765458422173 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272852920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57583328 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 2.5122472051249844E-4 systemCpuLoad: 0.02750910689611858 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:34:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:34:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272857920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57607800 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0011288097328483633 systemCpuLoad: 0.02044656297039639 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272862920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57607800 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271951831409935E-4 systemCpuLoad: 0.019816881976671266 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272867920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57607800 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.272738677706687E-4 systemCpuLoad: 0.020326223337515686 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272872920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57613504 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.787346221441124E-4 systemCpuLoad: 0.01845342706502636 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272877920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57613504 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.016302984700275E-4 systemCpuLoad: 0.019563581640331076 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272882920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57616552 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.528230865746549E-4 systemCpuLoad: 0.019696399447998995 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:34:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:35:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272887921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57644496 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010018785222291797 systemCpuLoad: 0.024799599198396794 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272892920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57652808 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.279040562602035E-4 systemCpuLoad: 0.02473320778405524 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272897920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57652808 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.532011046949536E-4 systemCpuLoad: 0.020341536916122553 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272902920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57658112 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010046464900163254 systemCpuLoad: 0.017830236062280264 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272907920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57658112 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.268020559107434E-4 systemCpuLoad: 0.017049015920772223 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272912920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57658112 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.030813734121495E-4 systemCpuLoad: 0.016220294228592985 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:35:13 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:35:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272917920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57680224 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0011302273012683663 systemCpuLoad: 0.0239919608089436 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272922920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57681472 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.013159543802481E-4 systemCpuLoad: 0.02218323098132598 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272927920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 75368600 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57681472 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.014416447285947E-4 systemCpuLoad: 0.021311269900965275 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272932920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57686896 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0012553351744915894 systemCpuLoad: 0.020087884494664157 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272937925 collectInterval: 5005 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57686896 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.270378730875344E-4 systemCpuLoad: 0.0234423968910618 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272942920 collectInterval: 4995 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57688432 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.272738677706687E-4 systemCpuLoad: 0.02082810539523212 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:35:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:36:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272947920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57663664 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010048988820499938 systemCpuLoad: 0.021356783919597992 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272952920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57663664 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.768844221105528E-4 systemCpuLoad: 0.02223897474557105 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272957920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57663664 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.531065645788879E-4 systemCpuLoad: 0.018574297188755022 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272962920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57584280 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001380002509095471 systemCpuLoad: 0.024341279799247176 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272967920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 78699584 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57584280 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.019450370184465E-4 systemCpuLoad: 0.02082810539523212 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272972920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57584280 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.2758880381574E-4 systemCpuLoad: 0.020459395004393123 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:36:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:36:13 [DEBUG](c.n.p.c.r.g.s.AgentService         ) Request PAgentInfo=hostname: "563482ed4875" ip: "172.24.0.12" serviceType: 1010 pid: 1 agentVersion: "2.0.1" vmVersion: "1.8.0_171" serverMetaData { vmArg: "-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties" vmArg: "-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" vmArg: "-javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar" vmArg: "-Dpinpoint.agentId=app-in-docker" vmArg: "-Dpinpoint.applicationName=quickapp" vmArg: "-Dpinpoint.profiler.profiles.active=release" vmArg: "-Djdk.tls.ephemeralDHKeySize=2048" vmArg: "-Djava.protocol.handler.pkgs=org.apache.catalina.webresources" vmArg: "-Dorg.apache.catalina.security.SecurityListener.UMASK=0027" vmArg: "-Dignore.endorsed.dirs=" vmArg: "-Dcatalina.base=/usr/local/tomcat" vmArg: "-Dcatalina.home=/usr/local/tomcat" vmArg: "-Djava.io.tmpdir=/usr/local/tomcat/temp" serviceInfo { serviceName: "Catalina/localhost/" serviceLib: "spring-core-4.1.7.RELEASE.jar" serviceLib: "spring-web-4.1.7.RELEASE.jar" serviceLib: "slf4j-api-1.7.5.jar" serviceLib: "log4j-1.2.17.jar" serviceLib: "commons-io-2.4.jar" serviceLib: "jackson-annotations-2.4.0.jar" serviceLib: "spring-webmvc-4.1.7.RELEASE.jar" serviceLib: "jcl-over-slf4j-1.7.5.jar" serviceLib: "jackson-core-2.4.4.jar" serviceLib: "jackson-databind-2.4.4.jar" serviceLib: "jstl-impl-1.2.jar" serviceLib: "commons-codec-1.6.jar" serviceLib: "httpclient-4.3.6.jar" serviceLib: "httpcore-4.3.3.jar" serviceLib: "aopalliance-1.0.jar" serviceLib: "commons-logging-1.1.3.jar" serviceLib: "spring-beans-4.1.7.RELEASE.jar" serviceLib: "spring-context-4.1.7.RELEASE.jar" serviceLib: "jstl-api-1.2.jar" serviceLib: "spring-aop-4.1.7.RELEASE.jar" serviceLib: "slf4j-log4j12-1.7.5.jar" serviceLib: "spring-expression-4.1.7.RELEASE.jar" serviceLib: "standard-1.1.2.jar" } } jvmInfo { vmVersion: "1.8.0_171" gcType: JVM_GC_TYPE_PARALLEL }
2020-03-27 01:36:13 [DEBUG](c.n.p.c.d.h.HbaseAgentInfoDao      ) insert agent info. AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:13 [DEBUG](.n.p.c.d.h.HbaseApplicationIndexDao) Insert ApplicationIndex: AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:16 [DEBUG](c.n.p.c.r.g.s.AgentService         ) Request PAgentInfo=hostname: "563482ed4875" ip: "172.24.0.12" serviceType: 1010 pid: 1 agentVersion: "2.0.1" vmVersion: "1.8.0_171" serverMetaData { vmArg: "-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties" vmArg: "-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" vmArg: "-javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar" vmArg: "-Dpinpoint.agentId=app-in-docker" vmArg: "-Dpinpoint.applicationName=quickapp" vmArg: "-Dpinpoint.profiler.profiles.active=release" vmArg: "-Djdk.tls.ephemeralDHKeySize=2048" vmArg: "-Djava.protocol.handler.pkgs=org.apache.catalina.webresources" vmArg: "-Dorg.apache.catalina.security.SecurityListener.UMASK=0027" vmArg: "-Dignore.endorsed.dirs=" vmArg: "-Dcatalina.base=/usr/local/tomcat" vmArg: "-Dcatalina.home=/usr/local/tomcat" vmArg: "-Djava.io.tmpdir=/usr/local/tomcat/temp" serviceInfo { serviceName: "Catalina/localhost/" serviceLib: "spring-core-4.1.7.RELEASE.jar" serviceLib: "spring-web-4.1.7.RELEASE.jar" serviceLib: "slf4j-api-1.7.5.jar" serviceLib: "log4j-1.2.17.jar" serviceLib: "commons-io-2.4.jar" serviceLib: "jackson-annotations-2.4.0.jar" serviceLib: "spring-webmvc-4.1.7.RELEASE.jar" serviceLib: "jcl-over-slf4j-1.7.5.jar" serviceLib: "jackson-core-2.4.4.jar" serviceLib: "jackson-databind-2.4.4.jar" serviceLib: "jstl-impl-1.2.jar" serviceLib: "commons-codec-1.6.jar" serviceLib: "httpclient-4.3.6.jar" serviceLib: "httpcore-4.3.3.jar" serviceLib: "aopalliance-1.0.jar" serviceLib: "commons-logging-1.1.3.jar" serviceLib: "spring-beans-4.1.7.RELEASE.jar" serviceLib: "spring-context-4.1.7.RELEASE.jar" serviceLib: "jstl-api-1.2.jar" serviceLib: "spring-aop-4.1.7.RELEASE.jar" serviceLib: "slf4j-log4j12-1.7.5.jar" serviceLib: "spring-expression-4.1.7.RELEASE.jar" serviceLib: "standard-1.1.2.jar" } } jvmInfo { vmVersion: "1.8.0_171" gcType: JVM_GC_TYPE_PARALLEL }
2020-03-27 01:36:16 [DEBUG](c.n.p.c.d.h.HbaseAgentInfoDao      ) insert agent info. AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:16 [DEBUG](.n.p.c.d.h.HbaseApplicationIndexDao) Insert ApplicationIndex: AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:16 [DEBUG](c.n.p.c.r.g.s.AgentService         ) Request PAgentInfo=hostname: "563482ed4875" ip: "172.24.0.12" serviceType: 1010 pid: 1 agentVersion: "2.0.1" vmVersion: "1.8.0_171" serverMetaData { vmArg: "-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties" vmArg: "-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" vmArg: "-javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar" vmArg: "-Dpinpoint.agentId=app-in-docker" vmArg: "-Dpinpoint.applicationName=quickapp" vmArg: "-Dpinpoint.profiler.profiles.active=release" vmArg: "-Djdk.tls.ephemeralDHKeySize=2048" vmArg: "-Djava.protocol.handler.pkgs=org.apache.catalina.webresources" vmArg: "-Dorg.apache.catalina.security.SecurityListener.UMASK=0027" vmArg: "-Dignore.endorsed.dirs=" vmArg: "-Dcatalina.base=/usr/local/tomcat" vmArg: "-Dcatalina.home=/usr/local/tomcat" vmArg: "-Djava.io.tmpdir=/usr/local/tomcat/temp" serviceInfo { serviceName: "Catalina/localhost/" serviceLib: "spring-core-4.1.7.RELEASE.jar" serviceLib: "spring-web-4.1.7.RELEASE.jar" serviceLib: "slf4j-api-1.7.5.jar" serviceLib: "log4j-1.2.17.jar" serviceLib: "commons-io-2.4.jar" serviceLib: "jackson-annotations-2.4.0.jar" serviceLib: "spring-webmvc-4.1.7.RELEASE.jar" serviceLib: "jcl-over-slf4j-1.7.5.jar" serviceLib: "jackson-core-2.4.4.jar" serviceLib: "jackson-databind-2.4.4.jar" serviceLib: "jstl-impl-1.2.jar" serviceLib: "commons-codec-1.6.jar" serviceLib: "httpclient-4.3.6.jar" serviceLib: "httpcore-4.3.3.jar" serviceLib: "aopalliance-1.0.jar" serviceLib: "commons-logging-1.1.3.jar" serviceLib: "spring-beans-4.1.7.RELEASE.jar" serviceLib: "spring-context-4.1.7.RELEASE.jar" serviceLib: "jstl-api-1.2.jar" serviceLib: "spring-aop-4.1.7.RELEASE.jar" serviceLib: "slf4j-log4j12-1.7.5.jar" serviceLib: "spring-expression-4.1.7.RELEASE.jar" serviceLib: "standard-1.1.2.jar" } } jvmInfo { vmVersion: "1.8.0_171" gcType: JVM_GC_TYPE_PARALLEL }
2020-03-27 01:36:16 [DEBUG](c.n.p.c.d.h.HbaseAgentInfoDao      ) insert agent info. AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:16 [DEBUG](.n.p.c.d.h.HbaseApplicationIndexDao) Insert ApplicationIndex: AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:17 [DEBUG](c.n.p.c.r.g.s.AgentService         ) Request PAgentInfo=hostname: "563482ed4875" ip: "172.24.0.12" serviceType: 1010 pid: 1 agentVersion: "2.0.1" vmVersion: "1.8.0_171" serverMetaData { vmArg: "-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties" vmArg: "-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager" vmArg: "-javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar" vmArg: "-Dpinpoint.agentId=app-in-docker" vmArg: "-Dpinpoint.applicationName=quickapp" vmArg: "-Dpinpoint.profiler.profiles.active=release" vmArg: "-Djdk.tls.ephemeralDHKeySize=2048" vmArg: "-Djava.protocol.handler.pkgs=org.apache.catalina.webresources" vmArg: "-Dorg.apache.catalina.security.SecurityListener.UMASK=0027" vmArg: "-Dignore.endorsed.dirs=" vmArg: "-Dcatalina.base=/usr/local/tomcat" vmArg: "-Dcatalina.home=/usr/local/tomcat" vmArg: "-Djava.io.tmpdir=/usr/local/tomcat/temp" serviceInfo { serviceName: "Catalina/localhost/" serviceLib: "spring-core-4.1.7.RELEASE.jar" serviceLib: "spring-web-4.1.7.RELEASE.jar" serviceLib: "slf4j-api-1.7.5.jar" serviceLib: "log4j-1.2.17.jar" serviceLib: "commons-io-2.4.jar" serviceLib: "jackson-annotations-2.4.0.jar" serviceLib: "spring-webmvc-4.1.7.RELEASE.jar" serviceLib: "jcl-over-slf4j-1.7.5.jar" serviceLib: "jackson-core-2.4.4.jar" serviceLib: "jackson-databind-2.4.4.jar" serviceLib: "jstl-impl-1.2.jar" serviceLib: "commons-codec-1.6.jar" serviceLib: "httpclient-4.3.6.jar" serviceLib: "httpcore-4.3.3.jar" serviceLib: "aopalliance-1.0.jar" serviceLib: "commons-logging-1.1.3.jar" serviceLib: "spring-beans-4.1.7.RELEASE.jar" serviceLib: "spring-context-4.1.7.RELEASE.jar" serviceLib: "jstl-api-1.2.jar" serviceLib: "spring-aop-4.1.7.RELEASE.jar" serviceLib: "slf4j-log4j12-1.7.5.jar" serviceLib: "spring-expression-4.1.7.RELEASE.jar" serviceLib: "standard-1.1.2.jar" } } jvmInfo { vmVersion: "1.8.0_171" gcType: JVM_GC_TYPE_PARALLEL }
2020-03-27 01:36:17 [DEBUG](c.n.p.c.d.h.HbaseAgentInfoDao      ) insert agent info. AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:17 [DEBUG](.n.p.c.d.h.HbaseApplicationIndexDao) Insert ApplicationIndex: AgentInfoBo{hostName='563482ed4875', ip='172.24.0.12', ports='', agentId='app-in-docker', applicationName='quickapp', serviceTypeCode=1010, pid=1, vmVersion=1.8.0_171', agentVersion='2.0.1', startTime=1585272660934, endTimeStamp=0, endStatus=0}
2020-03-27 01:36:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585272977920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57428096 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.004518072289156626 systemCpuLoad: 0.03438755020080321 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272982920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57428096 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.273525721455459E-4 systemCpuLoad: 0.018820577164366373 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272987921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57428992 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.7612838515546637E-4 systemCpuLoad: 0.01930792377131394 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272992920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57431312 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001005530417295123 systemCpuLoad: 0.03167420814479638 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585272997924 collectInterval: 5004 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57431312 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.7655328228944395E-4 systemCpuLoad: 0.021968365553602813 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273002920 collectInterval: 4996 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57431312 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.02071043052592E-4 systemCpuLoad: 0.016568344420735535 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:36:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:37:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273007920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57447048 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0012558081125204068 systemCpuLoad: 0.019213864121562223 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273012920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57447048 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.023232450081628E-4 systemCpuLoad: 0.022596033140848605 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273017920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57447048 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271165182490907E-4 systemCpuLoad: 0.019443050677370795 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273022920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57459096 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001005151400929765 systemCpuLoad: 0.0159607892421767 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273027920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57459096 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.022601707684581E-4 systemCpuLoad: 0.019839276745354094 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273032920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57459096 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.017561465127947E-4 systemCpuLoad: 0.019443050677370795 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:37:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:37:28 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/10.97.195.13:51864}
2020-03-27 01:37:28 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/10.97.195.13:51860}
2020-03-27 01:37:28 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/10.97.195.13:33206}
2020-03-27 01:37:28 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/10.97.195.13:39492}
2020-03-27 01:37:28 [DEBUG](c.n.p.c.r.g.s.c.GrpcCommandService ) AgentInfo{web-user-desa:web-user-desa-n1:1585273042156:} => local. handleCommand(). transportId:4
2020-03-27 01:37:28 [INFO ](c.n.p.c.r.g.s.c.GrpcCommandService ) web-user-desa:web-user-desa-n1:1585273042156 => local. ready() transportId:4
2020-03-27 01:37:28 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:CONNECTED ,before:NONE ,current:CONNECTED).
2020-03-27 01:37:28 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 1 apiInfo: "Asynchronous Invocation" line: -1 type: 200
2020-03-27 01:37:28 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=1, apiInfo='Asynchronous Invocation', lineNumber=-1, methodTypeEnum=INVOCATION}
2020-03-27 01:37:34 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 2 apiInfo: "org.postgresql.Driver.connect(java.lang.String url, java.util.Properties info)" line: 211
2020-03-27 01:37:34 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=2, apiInfo='org.postgresql.Driver.connect(java.lang.String url, java.util.Properties info)', lineNumber=211, methodTypeEnum=DEFAULT}
2020-03-27 01:37:35 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 4 apiInfo: "org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int autoGeneratedKeys)" line: 1693
2020-03-27 01:37:35 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 6 apiInfo: "org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int resultSetType, int resultSetConcurrency)" line: 1680
2020-03-27 01:37:35 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 8 apiInfo: "org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, java.lang.String[] columnNames)" line: 1713
2020-03-27 01:37:35 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=4, apiInfo='org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int autoGeneratedKeys)', lineNumber=1693, methodTypeEnum=DEFAULT}
2020-03-27 01:37:35 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=6, apiInfo='org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int resultSetType, int resultSetConcurrency)', lineNumber=1680, methodTypeEnum=DEFAULT}
2020-03-27 01:37:35 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 5 apiInfo: "org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int[] columnIndexes)" line: 1702
2020-03-27 01:37:35 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 7 apiInfo: "org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability)" line: 1222
2020-03-27 01:37:35 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=5, apiInfo='org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int[] columnIndexes)', lineNumber=1702, methodTypeEnum=DEFAULT}
2020-03-27 01:37:35 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=8, apiInfo='org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, java.lang.String[] columnNames)', lineNumber=1713, methodTypeEnum=DEFAULT}
2020-03-27 01:37:35 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=7, apiInfo='org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql, int resultSetType, int resultSetConcurrency, int resultSetHoldability)', lineNumber=1222, methodTypeEnum=DEFAULT}
2020-03-27 01:37:35 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 3 apiInfo: "org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql)" line: 371
2020-03-27 01:37:35 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=3, apiInfo='org.postgresql.jdbc.PgConnection.prepareStatement(java.lang.String sql)', lineNumber=371, methodTypeEnum=DEFAULT}
2020-03-27 01:37:36 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 9 apiInfo: "org.postgresql.jdbc.PgStatement.executeQuery(java.lang.String p_sql)" line: 224
2020-03-27 01:37:36 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 10 apiInfo: "org.postgresql.jdbc.PgStatement.executeUpdate(java.lang.String p_sql)" line: 244
2020-03-27 01:37:36 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 11 apiInfo: "org.postgresql.jdbc.PgStatement.executeUpdate(java.lang.String sql, int autoGeneratedKeys)" line: 1138
2020-03-27 01:37:36 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 13 apiInfo: "org.postgresql.jdbc.PgStatement.execute(java.lang.String sql, int autoGeneratedKeys)" line: 1167
2020-03-27 01:37:36 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=9, apiInfo='org.postgresql.jdbc.PgStatement.executeQuery(java.lang.String p_sql)', lineNumber=224, methodTypeEnum=DEFAULT}
2020-03-27 01:37:36 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=11, apiInfo='org.postgresql.jdbc.PgStatement.executeUpdate(java.lang.String sql, int autoGeneratedKeys)', lineNumber=1138, methodTypeEnum=DEFAULT}
2020-03-27 01:37:36 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=10, apiInfo='org.postgresql.jdbc.PgStatement.executeUpdate(java.lang.String p_sql)', lineNumber=244, methodTypeEnum=DEFAULT}
2020-03-27 01:37:36 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=13, apiInfo='org.postgresql.jdbc.PgStatement.execute(java.lang.String sql, int autoGeneratedKeys)', lineNumber=1167, methodTypeEnum=DEFAULT}
2020-03-27 01:37:36 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 12 apiInfo: "org.postgresql.jdbc.PgStatement.execute(java.lang.String p_sql)" line: 266
2020-03-27 01:37:36 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=12, apiInfo='org.postgresql.jdbc.PgStatement.execute(java.lang.String p_sql)', lineNumber=266, methodTypeEnum=DEFAULT}
2020-03-27 01:37:39 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 14 apiInfo: "java.util.concurrent.CompletableFuture.get()" line: 1895
2020-03-27 01:37:39 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 16 apiInfo: "java.util.concurrent.CompletableFuture.join()" line: 1934
2020-03-27 01:37:39 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=14, apiInfo='java.util.concurrent.CompletableFuture.get()', lineNumber=1895, methodTypeEnum=DEFAULT}
2020-03-27 01:37:39 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=16, apiInfo='java.util.concurrent.CompletableFuture.join()', lineNumber=1934, methodTypeEnum=DEFAULT}
2020-03-27 01:37:39 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 18 apiInfo: "java.util.concurrent.CompletableFuture.cancel(boolean boolean)" line: 2263
2020-03-27 01:37:39 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 15 apiInfo: "java.util.concurrent.CompletableFuture.get(long long, java.util.concurrent.TimeUnit TimeUnit)" line: 1914
2020-03-27 01:37:39 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 17 apiInfo: "java.util.concurrent.CompletableFuture.complete(java.lang.Object Object)" line: 1961
2020-03-27 01:37:39 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=18, apiInfo='java.util.concurrent.CompletableFuture.cancel(boolean boolean)', lineNumber=2263, methodTypeEnum=DEFAULT}
2020-03-27 01:37:39 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=15, apiInfo='java.util.concurrent.CompletableFuture.get(long long, java.util.concurrent.TimeUnit TimeUnit)', lineNumber=1914, methodTypeEnum=DEFAULT}
2020-03-27 01:37:39 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=17, apiInfo='java.util.concurrent.CompletableFuture.complete(java.lang.Object Object)', lineNumber=1961, methodTypeEnum=DEFAULT}
2020-03-27 01:37:41 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 19 apiInfo: "io.undertow.server.Connectors.executeRootHandler(io.undertow.server.HttpHandler handler, io.undertow.server.HttpServerExchange exchange)" line: 201
2020-03-27 01:37:41 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=19, apiInfo='io.undertow.server.Connectors.executeRootHandler(io.undertow.server.HttpHandler handler, io.undertow.server.HttpServerExchange exchange)', lineNumber=201, methodTypeEnum=DEFAULT}
2020-03-27 01:37:41 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 20 apiInfo: "Servlet Process" line: -1 type: 100
2020-03-27 01:37:41 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=20, apiInfo='Servlet Process', lineNumber=-1, methodTypeEnum=WEB_REQUEST}
2020-03-27 01:37:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273037921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57464608 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.788449466415569E-4 systemCpuLoad: 0.019708762239517952 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273042921 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57464608 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.785140562248996E-4 systemCpuLoad: 0.018190942165349392 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273047920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57466552 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.027020233756441E-4 systemCpuLoad: 0.026269482151835093 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273052920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57471160 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.525398218989089E-4 systemCpuLoad: 0.02220549491908167 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273057922 collectInterval: 5002 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57471160 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.270378730875344E-4 systemCpuLoad: 0.03097955600150508 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273062920 collectInterval: 4998 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57475320 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.550018875047187E-4 systemCpuLoad: 0.023782559456398643 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:37:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:37:55 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 21 apiInfo: "org.postgresql.jdbc.PgPreparedStatement.execute()" line: 132
2020-03-27 01:37:55 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 22 apiInfo: "org.postgresql.jdbc.PgPreparedStatement.executeQuery()" line: 106
2020-03-27 01:37:55 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 23 apiInfo: "org.postgresql.jdbc.PgPreparedStatement.executeUpdate()" line: 120
2020-03-27 01:37:55 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=21, apiInfo='org.postgresql.jdbc.PgPreparedStatement.execute()', lineNumber=132, methodTypeEnum=DEFAULT}
2020-03-27 01:37:55 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=22, apiInfo='org.postgresql.jdbc.PgPreparedStatement.executeQuery()', lineNumber=106, methodTypeEnum=DEFAULT}
2020-03-27 01:37:55 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=23, apiInfo='org.postgresql.jdbc.PgPreparedStatement.executeUpdate()', lineNumber=120, methodTypeEnum=DEFAULT}
2020-03-27 01:37:55 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 24 apiInfo: "com.squareup.okhttp.Dispatcher.cancel(java.lang.Object tag)" line: 118
2020-03-27 01:37:55 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=24, apiInfo='com.squareup.okhttp.Dispatcher.cancel(java.lang.Object tag)', lineNumber=118, methodTypeEnum=DEFAULT}
2020-03-27 01:37:55 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 25 apiInfo: "com.squareup.okhttp.Dispatcher.enqueue(com.squareup.okhttp.Call$AsyncCall call)" line: 108
2020-03-27 01:37:55 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=25, apiInfo='com.squareup.okhttp.Dispatcher.enqueue(com.squareup.okhttp.Call$AsyncCall call)', lineNumber=108, methodTypeEnum=DEFAULT}
2020-03-27 01:37:55 [DEBUG](c.n.p.c.r.g.s.MetadataService      ) Request PApiMetaData=apiId: 26 apiInfo: "com.google.gson.Gson.fromJson(java.lang.String json, java.lang.Class classOfT)" line: 773
2020-03-27 01:37:55 [DEBUG](c.n.p.c.d.h.HbaseApiMetaDataDao    ) insert:ApiMetaDataBo{agentId='web-user-desa-n1', startTime=1585273042156, apiId=26, apiInfo='com.google.gson.Gson.fromJson(java.lang.String json, java.lang.Class classOfT)', lineNumber=773, methodTypeEnum=DEFAULT}
2020-03-27 01:37:58 [DEBUG](r.g.PermissionServerTransportFilter) Terminated attributes={transportMetadata=DefaultTransportMetadata{debugString='grpcAgentReceiver', remoteAddress=/172.24.0.12:33146, transportId=0, connectTime=1585272718344}, remote-addr=/172.24.0.12:33146}
2020-03-27 01:37:58 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273053147 collectInterval: 5031 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 390350216 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 74092536 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.6359539308963446 systemCpuLoad: 0.8057085628442664 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 518 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273058147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 575023984 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 100821760 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.5326633165829145 systemCpuLoad: 0.7301507537688443 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 569 } directBuffer { directCount: 3 directMemoryUsed: 368641 } } agentStat { timestamp: 1585273063147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 281577584 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 120919352 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.40191628845184063 systemCpuLoad: 0.5642965204236006 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 691 } directBuffer { directCount: 21 directMemoryUsed: 118876284 } } agentStat { timestamp: 1585273068147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 339149120 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 128000552 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.12927565392354123 systemCpuLoad: 0.3345070422535211 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 716 } directBuffer { directCount: 26 directMemoryUsed: 118954097 } } agentStat { timestamp: 1585273073150 collectInterval: 5003 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 514728576 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 133297528 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.2974874371859297 systemCpuLoad: 0.5246231155778894 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 741 } directBuffer { directCount: 28 directMemoryUsed: 118970481 } } agentStat { timestamp: 1585273078448 collectInterval: 5298 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 173213328 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 171652632 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.7683562292752251 systemCpuLoad: 0.916153481762198 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 26 directMemoryUsed: 118970442 } } }
2020-03-27 01:37:58 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:38:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273067920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57481912 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.533902561526872E-4 systemCpuLoad: 0.01757689893283114 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273072921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57481912 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.268020559107434E-4 systemCpuLoad: 0.026827127992979816 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273077922 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 80398464 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57481912 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.53012048192771E-4 systemCpuLoad: 0.02698293172690763 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 230 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273082920 collectInterval: 4998 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57505480 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0012559658377292136 systemCpuLoad: 0.02688104509483733 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273087920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57505480 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.021971123666039E-4 systemCpuLoad: 0.023594377510040163 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273092920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57506184 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.533902561526872E-4 systemCpuLoad: 0.01733450571536239 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:38:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:38:28 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273083147 collectInterval: 4699 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 342308648 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 185962344 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.44711538461538464 systemCpuLoad: 0.642094017094017 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273088147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 343579632 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 185981416 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015067805123053744 systemCpuLoad: 0.16725263686589653 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273093147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 354233088 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 186993344 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.03560682046138415 systemCpuLoad: 0.1905717151454363 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273098147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 354897200 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187006496 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.002004008016032064 systemCpuLoad: 0.20691382765531063 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273103147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 356098648 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187037936 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0025075225677031092 systemCpuLoad: 0.15145436308926782 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273108147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 360832520 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187057744 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0030105368790767687 systemCpuLoad: 0.16056196688409433 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } }
2020-03-27 01:38:28 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:38:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273097920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57506072 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010047726701833712 systemCpuLoad: 0.020595253045334674 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273102920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57506072 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271165182490907E-4 systemCpuLoad: 0.01755926251097454 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273107920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57504304 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.025125628140704E-4 systemCpuLoad: 0.017839195979899497 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273112920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57514024 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0010030090270812437 systemCpuLoad: 0.022063432368058168 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273117920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57514024 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.533902561526871E-4 systemCpuLoad: 0.020846414667838757 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273122920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57514024 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.270378730875344E-4 systemCpuLoad: 0.017924291802456756 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:38:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:38:58 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273113147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 374139784 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187109216 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0035158211953792064 systemCpuLoad: 0.20894023103967857 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273118147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 374704264 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187159320 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020060180541624875 systemCpuLoad: 0.15797392176529587 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 28 directMemoryUsed: 119019594 } } agentStat { timestamp: 1585273123147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375036152 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187205984 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.004016064257028112 systemCpuLoad: 0.15110441767068272 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273128147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375036152 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187150080 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020050125313283208 systemCpuLoad: 0.20651629072681704 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273133147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375036152 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187195136 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0025075225677031092 systemCpuLoad: 0.15897693079237715 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273138147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375239304 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187241528 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.003009027081243731 systemCpuLoad: 0.1715145436308927 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:38:58 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:38:58 [INFO ](c.n.p.c.r.g.s.AgentService         ) Failed to ping stream, id=0, cause=CANCELLED: cancelled before receiving half close
2020-03-27 01:38:58 [INFO ](c.n.p.c.r.g.s.c.GrpcCommandService ) Failed to command stream, web-user-desa:web-user-desa-n1:1585273042156 => local, cause=CANCELLED: cancelled before receiving half close
2020-03-27 01:38:58 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:ERROR_ILLEGAL_STATE_CHANGE ,before:CONNECTED ,current:ERROR_ILLEGAL_STATE_CHANGE).
2020-03-27 01:38:58 [WARN ](c.n.p.c.r.g.PinpointGrpcServer     ) stop(). Socket has unexpected state(CONNECTED)
2020-03-27 01:38:58 [INFO ](c.n.p.c.r.g.PinpointGrpcServer     ) web-user-desa:web-user-desa-n1:1585273042156 <=> local all streamChannels will be close.
2020-03-27 01:39:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273127920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57515944 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001005530417295123 systemCpuLoad: 0.020249025279839013 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273132920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57515944 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.015673981191223E-4 systemCpuLoad: 0.020940438871473357 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273137920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57515944 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.800603469952225E-4 systemCpuLoad: 0.017726929846618052 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273142920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57483560 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.779631255487271E-4 systemCpuLoad: 0.022071733132681216 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273147924 collectInterval: 5004 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57483560 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.2672348959639E-4 systemCpuLoad: 0.020183026200325936 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273152920 collectInterval: 4996 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57483560 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.021340697966357E-4 systemCpuLoad: 0.016568344420735535 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:39:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:39:26 [DEBUG](r.g.PermissionServerTransportFilter) Terminated attributes={transportMetadata=DefaultTransportMetadata{debugString='grpcAgentReceiver', remoteAddress=/10.97.195.13:51860, transportId=5, connectTime=1585273048481}, remote-addr=/10.97.195.13:51860}
2020-03-27 01:39:28 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273143147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375368096 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187266240 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.002007024586051179 systemCpuLoad: 0.193176116407426 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273148147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375368096 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187285200 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015052684395383841 systemCpuLoad: 0.15755143000501756 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273153147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375496832 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187362520 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.002506265664160401 systemCpuLoad: 0.21253132832080202 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273158147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375496832 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187402048 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020050125313283208 systemCpuLoad: 0.1593984962406015 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273163147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375496832 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187409840 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015037593984962407 systemCpuLoad: 0.1619047619047619 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 746 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273168147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 375496832 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187425944 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020060180541624875 systemCpuLoad: 0.2021063189568706 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:39:28 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:39:29 [DEBUG](r.g.PermissionServerTransportFilter) Ready attributes={remote-addr=/10.97.195.13:51922}
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.s.c.GrpcCommandService ) AgentInfo{web-user-desa:web-user-desa-n1:1585273042156:} => local. handleCommand(). transportId:8
2020-03-27 01:39:29 [INFO ](c.n.p.c.r.g.s.c.GrpcCommandService ) web-user-desa:web-user-desa-n1:1585273042156 => local. ready() transportId:8
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:CONNECTED ,before:NONE ,current:CONNECTED).
2020-03-27 01:39:29 [INFO ](c.n.p.c.r.g.s.c.GrpcCommandService ) web-user-desa:web-user-desa-n1:1585273042156 => local. execute handshake:[740, 710, 730, 750]
2020-03-27 01:39:29 [INFO ](c.n.p.c.r.g.PinpointGrpcServer     ) AgentInfo{web-user-desa:web-user-desa-n1:1585273042156:} handleHandshake() started. data:[740, 710, 730, 750]
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:RUN_WITHOUT_HANDSHAKE ,before:CONNECTED ,current:RUN_WITHOUT_HANDSHAKE).
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.PinpointGrpcServer     ) Socket state change success(updateWanted:RUN_DUPLEX ,before:RUN_WITHOUT_HANDSHAKE ,current:RUN_DUPLEX).
2020-03-27 01:39:29 [DEBUG](c.n.p.c.c.z.ZookeeperJobWorker     ) addPinpointServer key:web-user-desa:web-user-desa-n1:1585273042156
2020-03-27 01:39:29 [DEBUG](c.n.p.c.c.z.ZookeeperJobWorker     ) handleUpdate zookeeperJobList:[ZookeeperJob{type=ADD, key='web-user-desa:web-user-desa-n1:1585273042156'}]
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.s.AgentService         ) PingSession:1 start:
2020-03-27 01:39:29 [INFO ](.n.p.c.r.g.s.AgentLifecycleListener) connect:PingSession{id=8, header=Header{agentId='web-user-desa-n1', applicationName='web-user-desa', agentStartTime=1585273042156, socketId=2}, eventIdAllocator=0}
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.s.KeepAliveService     ) updateState:PingSession{id=8, header=Header{agentId='web-user-desa-n1', applicationName='web-user-desa', agentStartTime=1585273042156, socketId=2}, eventIdAllocator=0} false Running/Agent connected
2020-03-27 01:39:29 [DEBUG](c.n.p.c.r.g.s.AgentService         ) PingSession:1 onNext:
2020-03-27 01:39:29 [DEBUG](.s.a.AgentLifeCycleAsyncTaskService) Handle lifecycle event - pinpointServer:DefaultAgentProperty{agentId='web-user-desa-n1', agentStartTime=1585273042156, properties={}}, state:Running
2020-03-27 01:39:29 [DEBUG](c.n.p.c.d.h.HbaseAgentLifeCycleDao ) insert agent life cycle. AgentLifeCycleBo{version=0, agentId='web-user-desa-n1', startTimestamp=1585273042156, eventTimestamp=1585273169457, eventIdentifier=8589934593, state=Running
2020-03-27 01:39:29 [DEBUG](c.n.p.c.s.AgentEventService        ) Insert AgentEventBo{version=1, agentId='web-user-desa-n1', startTimestamp=1585273042156, eventTimestamp=1585273169457, eventType='Agent connected'
2020-03-27 01:39:29 [DEBUG](c.n.p.c.d.h.HbaseAgentEventDao     ) insert agent event: AgentEventBo{version=1, agentId='web-user-desa-n1', startTimestamp=1585273042156, eventTimestamp=1585273169457, eventType='Agent connected'
2020-03-27 01:39:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273157920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57499264 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0012523481527864746 systemCpuLoad: 0.023046092184368736 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273162920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57499264 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.02008032128514E-4 systemCpuLoad: 0.017944535073409464 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273167920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57499264 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.276675872457947E-4 systemCpuLoad: 0.021463537090498305 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273172920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57389896 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.00100363818843307 systemCpuLoad: 0.030614805520702636 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273177920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57311480 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.526342197691921E-4 systemCpuLoad: 0.01806322127446061 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273182920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57311480 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.011275369581559E-4 systemCpuLoad: 0.019295827590527503 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:39:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:39:58 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273173147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 379735464 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187709672 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.009040683073832245 systemCpuLoad: 0.1747865394274234 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273178147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 379735464 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187716368 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015045135406218657 systemCpuLoad: 0.1489468405215647 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273183147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 379864200 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 187753224 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.002508780732563974 systemCpuLoad: 0.20421475163070746 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273188147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188132152 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.004513540621865597 systemCpuLoad: 0.16240601503759397 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273193147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188153176 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015052684395383843 systemCpuLoad: 0.161144578313253 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273198147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188179640 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015037593984962407 systemCpuLoad: 0.1899749373433584 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:39:58 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:40:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273187920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57230416 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0015088645794039985 systemCpuLoad: 0.019233186675047143 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273192920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57230416 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271951831409935E-4 systemCpuLoad: 0.02685069008782936 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273197920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57231120 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.787346221441124E-4 systemCpuLoad: 0.01870214635370905 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273202920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57241768 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.781834148789362E-4 systemCpuLoad: 0.018434913468773514 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273207920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57241768 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.7650602409638556E-4 systemCpuLoad: 0.01770467101958815 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273212920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57246888 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001254075746175069 systemCpuLoad: 0.018434913468773514 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:40:13 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:40:28 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273203153 collectInterval: 5006 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188155216 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.003006012024048096 systemCpuLoad: 0.16482965931863727 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273208147 collectInterval: 4994 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188155272 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0010045203415369162 systemCpuLoad: 0.19236564540431944 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273213147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188117448 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0025100401606425703 systemCpuLoad: 0.16516064257028112 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273218147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188120904 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.001506024096385542 systemCpuLoad: 0.1606425702811245 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273223147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188121608 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0010025062656641604 systemCpuLoad: 0.21954887218045113 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273228147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188154816 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.00501002004008016 systemCpuLoad: 0.14478957915831664 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:40:28 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:40:29 [DEBUG](c.n.p.c.r.g.s.AgentService         ) PingSession:1 onNext:
2020-03-27 01:40:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273217920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57289792 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0012556504269211452 systemCpuLoad: 0.020090406830738324 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273222920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57289792 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.766005523474768E-4 systemCpuLoad: 0.013055485814712529 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273227920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57292312 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.532011046949536E-4 systemCpuLoad: 0.01681937994226183 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273232920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57339136 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001003890074036893 systemCpuLoad: 0.0356425702811245 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273237920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57339136 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.287726358148893E-4 systemCpuLoad: 0.014079195474544313 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273242920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57340208 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.526342197691921E-4 systemCpuLoad: 0.021580928481806774 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:40:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:40:58 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273233147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188227920 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0035158211953792064 systemCpuLoad: 0.16122551481667505 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273238147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188245224 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0010035122930255895 systemCpuLoad: 0.2007024586051179 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273243147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188251416 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020060180541624875 systemCpuLoad: 0.15546639919759278 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273248147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188257688 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0010035122930255895 systemCpuLoad: 0.16407425990968388 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273253147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188279432 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015037593984962407 systemCpuLoad: 0.1974937343358396 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273258147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382071440 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188307840 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0040100250626566416 systemCpuLoad: 0.16140350877192983 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:40:58 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:41:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273247920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57376136 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.001257071024512885 systemCpuLoad: 0.019861722187303584 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273252920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57376136 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.265664160401002E-4 systemCpuLoad: 0.024313823787442033 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273257920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57376136 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.274312962730581E-4 systemCpuLoad: 0.023462986198243413 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273262920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57400872 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.529175555276697E-4 systemCpuLoad: 0.01794453507340946 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273267920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57400872 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.027020233756441E-4 systemCpuLoad: 0.01947970340580621 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273272920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57400872 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.50938673341677E-4 systemCpuLoad: 0.01839569515705168 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:41:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:41:28 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273263147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188523992 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.004014049172102358 systemCpuLoad: 0.19468138484696437 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273268147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188531392 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015045135406218657 systemCpuLoad: 0.17301905717151456 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273273147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188547480 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.003009027081243731 systemCpuLoad: 0.1584754262788365 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273278147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188557464 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020050125313283208 systemCpuLoad: 0.2055137844611529 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273283147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188593152 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0020060180541624875 systemCpuLoad: 0.15546639919759278 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273288146 collectInterval: 4999 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188599632 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0010025062656641604 systemCpuLoad: 0.15639097744360902 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:41:28 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:41:29 [DEBUG](c.n.p.c.r.g.s.AgentService         ) PingSession:1 onNext:
2020-03-27 01:41:42 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273277920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57408424 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.0013831258644536651 systemCpuLoad: 0.022887323943661973 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273282920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57408424 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.7707390648567115E-4 systemCpuLoad: 0.03984414278531925 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273287922 collectInterval: 5002 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57408424 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 5.019450370184465E-4 systemCpuLoad: 0.02547371062868616 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273292920 collectInterval: 4998 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57416808 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 7.53012048192771E-4 systemCpuLoad: 0.029987452948557092 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273297920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57416808 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.284565108094519E-4 systemCpuLoad: 0.022881569021875787 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273302920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57428848 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271951831409935E-4 systemCpuLoad: 0.01906912558022833 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:41:42 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:41:52 [DEBUG](c.n.p.c.r.g.s.KeepAliveService     ) updateState:PingSession{id=8, header=Header{agentId='web-user-desa-n1', applicationName='web-user-desa', agentStartTime=1585273042156, socketId=2}, eventIdAllocator=1} false Running/Agent ping
2020-03-27 01:41:52 [DEBUG](.s.a.AgentLifeCycleAsyncTaskService) Handle lifecycle event - pinpointServer:DefaultAgentProperty{agentId='web-user-desa-n1', agentStartTime=1585273042156, properties={}}, state:Running
2020-03-27 01:41:52 [DEBUG](c.n.p.c.d.h.HbaseAgentLifeCycleDao ) insert agent life cycle. AgentLifeCycleBo{version=0, agentId='web-user-desa-n1', startTimestamp=1585273042156, eventTimestamp=1585273312255, eventIdentifier=8589934594, state=Running
2020-03-27 01:41:52 [DEBUG](c.n.p.c.s.AgentEventService        ) Insert AgentEventBo{version=1, agentId='web-user-desa-n1', startTimestamp=1585273042156, eventTimestamp=1585273312255, eventType='Agent ping'
2020-03-27 01:41:52 [DEBUG](c.n.p.c.d.h.HbaseAgentEventDao     ) insert agent event: AgentEventBo{version=1, agentId='web-user-desa-n1', startTimestamp=1585273042156, eventTimestamp=1585273312255, eventType='Agent ping'
2020-03-27 01:41:58 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273293147 collectInterval: 5001 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188635640 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0025075225677031092 systemCpuLoad: 0.2086258776328987 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273298146 collectInterval: 4999 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188635640 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.001002004008016032 systemCpuLoad: 0.155310621242485 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273303147 collectInterval: 5001 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188678400 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.002007024586051179 systemCpuLoad: 0.14601103863522327 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273308146 collectInterval: 4999 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188683456 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.0015045135406218655 systemCpuLoad: 0.30641925777331996 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273313147 collectInterval: 5001 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382200176 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188833104 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.005513784461152882 systemCpuLoad: 0.15789473684210525 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } agentStat { timestamp: 1585273318147 collectInterval: 5000 gc { type: JVM_GC_TYPE_CMS jvmMemoryHeapUsed: 382457736 jvmMemoryHeapMax: 2114977792 jvmMemoryNonHeapUsed: 188873280 jvmMemoryNonHeapMax: 1879048192 } cpuLoad { jvmCpuLoad: 0.002007024586051179 systemCpuLoad: 0.16859006522829903 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 745 } directBuffer { directCount: 29 directMemoryUsed: 135796810 } } }
2020-03-27 01:41:58 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
2020-03-27 01:42:12 [DEBUG](c.n.p.c.r.g.s.StatService          ) Send PAgentStat=agentStatBatch { agentStat { timestamp: 1585273307920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57465768 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 0.00125517760763148 systemCpuLoad: 0.02372285678423497 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273312921 collectInterval: 5001 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57465768 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.28140703517588E-4 systemCpuLoad: 0.024748743718592967 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273317920 collectInterval: 4999 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57466344 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.285355122564425E-4 systemCpuLoad: 0.017219708396178986 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273322920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57479280 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 8.793969849246232E-4 systemCpuLoad: 0.017338861666038447 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273327920 collectInterval: 5000 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57479280 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 6.271951831409935E-4 systemCpuLoad: 0.01655795283492223 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } agentStat { timestamp: 1585273332928 collectInterval: 5008 gc { type: JVM_GC_TYPE_PARALLEL jvmMemoryHeapUsed: 81231264 jvmMemoryHeapMax: 7499415552 jvmMemoryNonHeapUsed: 57479280 jvmMemoryNonHeapMax: -1 jvmGcOldCount: 2 jvmGcOldTime: 264 } cpuLoad { jvmCpuLoad: 3.7556334501752625E-4 systemCpuLoad: 0.01740110165247872 } transaction { } activeTrace { histogram { histogramSchemaType: 2 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 activeTraceCount: 0 } } dataSourceList { } responseTime { } fileDescriptor { openFileDescriptorCount: 229 } directBuffer { directCount: 1 directMemoryUsed: 1 } } }
2020-03-27 01:42:12 [WARN ](c.n.p.c.s.SendAgentStatService     ) not send flink server. Because FlinkTcpDataSender is null.
`

Catalina.log

`27-Mar-2020 01:30:59.444 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [HTableFlushWorker-9] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
27-Mar-2020 01:30:59.490 INFO [Thread-10] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-9080"]
27-Mar-2020 01:30:59.503 INFO [Thread-10] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["ajp-nio-9009"]
27-Mar-2020 01:30:59.521 INFO [Thread-10] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-9080"]
27-Mar-2020 01:30:59.522 INFO [Thread-10] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["ajp-nio-9009"]
27-Mar-2020 01:31:07.436 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version:        Apache Tomcat/8.5.41
27-Mar-2020 01:31:07.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          May 4 2019 09:17:16 UTC
27-Mar-2020 01:31:07.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server number:         8.5.41.0
27-Mar-2020 01:31:07.444 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
27-Mar-2020 01:31:07.445 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            4.9.0-9-amd64
27-Mar-2020 01:31:07.445 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
27-Mar-2020 01:31:07.445 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-8-openjdk-amd64/jre
27-Mar-2020 01:31:07.445 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_212-8u212-b01-1~deb9u1-b01
27-Mar-2020 01:31:07.446 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
27-Mar-2020 01:31:07.446 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /usr/local/tomcat
27-Mar-2020 01:31:07.447 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/local/tomcat
27-Mar-2020 01:31:07.449 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
27-Mar-2020 01:31:07.449 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
27-Mar-2020 01:31:07.449 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dspring.profiles.active=release
27-Mar-2020 01:31:07.449 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
27-Mar-2020 01:31:07.450 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
27-Mar-2020 01:31:07.450 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
27-Mar-2020 01:31:07.453 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
27-Mar-2020 01:31:07.453 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/usr/local/tomcat
27-Mar-2020 01:31:07.454 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/local/tomcat
27-Mar-2020 01:31:07.455 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
27-Mar-2020 01:31:07.455 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded APR based Apache Tomcat Native library [1.2.21] using APR version [1.5.2].
27-Mar-2020 01:31:07.456 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
27-Mar-2020 01:31:07.457 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
27-Mar-2020 01:31:07.467 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.0j  20 Nov 2018]
27-Mar-2020 01:31:07.882 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-9080"]
27-Mar-2020 01:31:07.942 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
27-Mar-2020 01:31:08.024 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["ajp-nio-9009"]
27-Mar-2020 01:31:08.030 INFO [main] org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a shared selector for servlet write/read
27-Mar-2020 01:31:08.032 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 2498 ms
27-Mar-2020 01:31:08.126 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
27-Mar-2020 01:31:08.126 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet Engine: Apache Tomcat/8.5.41
27-Mar-2020 01:31:08.161 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/usr/local/tomcat/webapps/ROOT]
27-Mar-2020 01:31:25.604 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
27-Mar-2020 01:43:29.550 SEVERE [Pinpoint-GrpcAgent-Server-6] io.grpc.internal.SerializingExecutor.run Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@61c03bdb
        io.grpc.StatusRuntimeException: CANCELLED: call already cancelled
                at io.grpc.Status.asRuntimeException(Status.java:517)
                at io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:356)
                at com.navercorp.pinpoint.collector.receiver.grpc.PinpointGrpcServer.close(PinpointGrpcServer.java:289)
                at com.navercorp.pinpoint.collector.receiver.grpc.PinpointGrpcServer.disconnected(PinpointGrpcServer.java:273)
                at com.navercorp.pinpoint.collector.receiver.grpc.service.command.GrpcCommandService$2.onError(GrpcCommandService.java:131)
                at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:269)
                at io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40)
                at io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23)
                at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40)
                at io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96)
                at io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40)
                at io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23)
                at io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40)
                at io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96)
                at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:293)
                at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:738)
                at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
                at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
                at com.navercorp.pinpoint.collector.monitor.MonitoredRunnableDecorator$InstrumentedRunnable.run(MonitoredRunnableDecorator.java:113)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:748)
`

Agent.log

`2020-03-27 01:31:20 [WARN ](c.n.p.p.s.g.MetadataGrpcDataSender ) Error. request=apiId: 6 apiInfo: "org.springframework.web.servlet.FrameworkServlet.doPost(javax.servlet.http.HttpServletRequest request, javax.servlet.http.HttpServletResponse response)" line: 869, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9991
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:23 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:SpanStream-13
2020-03-27 01:31:23 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=SpanStream-13, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9993
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:23 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:StatStream-14
2020-03-27 01:31:23 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=StatStream-14, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9992
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:24 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-27 01:31:24 [WARN ](c.n.p.p.r.g.GrpcCommandService     ) Failed to command stream, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9991
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:24 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() started
2020-03-27 01:31:24 [INFO ](.p.r.g.GrpcActiveThreadCountService) close
2020-03-27 01:31:24 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() completed
2020-03-27 01:31:29 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:StatStream-15
2020-03-27 01:31:29 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=StatStream-15, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9992
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:30 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:SpanStream-16
2020-03-27 01:31:30 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=SpanStream-16, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9993
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:30 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-27 01:31:30 [WARN ](c.n.p.p.r.g.GrpcCommandService     ) Failed to command stream, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9991
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:30 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() started
2020-03-27 01:31:30 [INFO ](.p.r.g.GrpcActiveThreadCountService) close
2020-03-27 01:31:30 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() completed
2020-03-27 01:31:38 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:StatStream-17
2020-03-27 01:31:38 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=StatStream-17, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9992
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:40 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:SpanStream-18
2020-03-27 01:31:40 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=SpanStream-18, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9993
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:42 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-27 01:31:42 [WARN ](c.n.p.p.r.g.GrpcCommandService     ) Failed to command stream, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9991
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:42 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() started
2020-03-27 01:31:42 [INFO ](.p.r.g.GrpcActiveThreadCountService) close
2020-03-27 01:31:42 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() completed
2020-03-27 01:31:55 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:StatStream-19
2020-03-27 01:31:55 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=StatStream-19, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9992
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:31:57 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:SpanStream-20
2020-03-27 01:31:57 [WARN ](c.n.p.p.s.g.ResponseStreamObserver ) Failed to stream, name=SpanStream-20, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9993
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:32:00 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-27 01:32:00 [WARN ](c.n.p.p.r.g.GrpcCommandService     ) Failed to command stream, cause=UNAVAILABLE: io exception
io.grpc.StatusRuntimeException: UNAVAILABLE: io exception
        at io.grpc.Status.asRuntimeException(Status.java:526)
        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:420)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: pinpoint-collector/172.24.0.9:9991
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
        at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)
        at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:702)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-03-27 01:32:00 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() started
2020-03-27 01:32:00 [INFO ](.p.r.g.GrpcActiveThreadCountService) close
2020-03-27 01:32:00 [INFO ](c.n.p.p.r.g.GrpcCommandDispatcher  ) close() completed
2020-03-27 01:32:19 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:StatStream-21
2020-03-27 01:32:19 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) onReadyHandler:StatStream-21 eventNumber:0
2020-03-27 01:32:21 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) beforeStart:SpanStream-22
2020-03-27 01:32:21 [INFO ](c.n.p.p.s.g.ResponseStreamObserver ) onReadyHandler:SpanStream-22 eventNumber:0
2020-03-27 01:32:28 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Attempt to connect to CommandServiceStream.
2020-03-27 01:32:28 [INFO ](c.n.p.p.r.g.GrpcCommandService     ) Connect to CommandServiceStream completed.
2020-03-27 01:36:12 [INFO ](c.n.p.p.AgentInfoSender            ) Sending AgentInfo AgentInfo{agentInformation=DefaultAgentInformation{agentId='app-in-docker', applicationName='quickapp', isContainer=false, startTime=1585272660934, pid=1, machineName='563482ed4875', hostIp='172.24.0.12', serverType=TOMCAT, jvmVersion='1.8.0_171', agentVersion='2.0.1'}, serverMetaData=DefaultServerMetaData{serverInfo='', vmArgs=[-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties, -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, -javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar, -Dpinpoint.agentId=app-in-docker, -Dpinpoint.applicationName=quickapp, -Dpinpoint.profiler.profiles.active=release, -Djdk.tls.ephemeralDHKeySize=2048, -Djava.protocol.handler.pkgs=org.apache.catalina.webresources, -Dorg.apache.catalina.security.SecurityListener.UMASK=0027, -Dignore.endorsed.dirs=, -Dcatalina.base=/usr/local/tomcat, -Dcatalina.home=/usr/local/tomcat, -Djava.io.tmpdir=/usr/local/tomcat/temp], connectors={8080=HTTP/1.1, 8009=AJP/1.3}, serviceInfo=[DefaultServiceInfo{serviceName='Catalina/localhost/', serviceLibs=[spring-core-4.1.7.RELEASE.jar, spring-web-4.1.7.RELEASE.jar, slf4j-api-1.7.5.jar, log4j-1.2.17.jar, commons-io-2.4.jar, jackson-annotations-2.4.0.jar, spring-webmvc-4.1.7.RELEASE.jar, jcl-over-slf4j-1.7.5.jar, jackson-core-2.4.4.jar, jackson-databind-2.4.4.jar, jstl-impl-1.2.jar, commons-codec-1.6.jar, httpclient-4.3.6.jar, httpcore-4.3.3.jar, aopalliance-1.0.jar, commons-logging-1.1.3.jar, spring-beans-4.1.7.RELEASE.jar, spring-context-4.1.7.RELEASE.jar, jstl-api-1.2.jar, spring-aop-4.1.7.RELEASE.jar, slf4j-log4j12-1.7.5.jar, spring-expression-4.1.7.RELEASE.jar, standard-1.1.2.jar]}]}, jvmInformation=com.navercorp.pinpoint.profiler.JvmInformation@20403f57}
2020-03-27 01:36:13 [INFO ](c.n.p.p.AgentInfoSender            ) AgentInfo sent.
2020-03-27 01:36:16 [INFO ](c.n.p.p.AgentInfoSender            ) Sending AgentInfo AgentInfo{agentInformation=DefaultAgentInformation{agentId='app-in-docker', applicationName='quickapp', isContainer=false, startTime=1585272660934, pid=1, machineName='563482ed4875', hostIp='172.24.0.12', serverType=TOMCAT, jvmVersion='1.8.0_171', agentVersion='2.0.1'}, serverMetaData=DefaultServerMetaData{serverInfo='', vmArgs=[-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties, -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, -javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar, -Dpinpoint.agentId=app-in-docker, -Dpinpoint.applicationName=quickapp, -Dpinpoint.profiler.profiles.active=release, -Djdk.tls.ephemeralDHKeySize=2048, -Djava.protocol.handler.pkgs=org.apache.catalina.webresources, -Dorg.apache.catalina.security.SecurityListener.UMASK=0027, -Dignore.endorsed.dirs=, -Dcatalina.base=/usr/local/tomcat, -Dcatalina.home=/usr/local/tomcat, -Djava.io.tmpdir=/usr/local/tomcat/temp], connectors={8080=HTTP/1.1, 8009=AJP/1.3}, serviceInfo=[DefaultServiceInfo{serviceName='Catalina/localhost/', serviceLibs=[spring-core-4.1.7.RELEASE.jar, spring-web-4.1.7.RELEASE.jar, slf4j-api-1.7.5.jar, log4j-1.2.17.jar, commons-io-2.4.jar, jackson-annotations-2.4.0.jar, spring-webmvc-4.1.7.RELEASE.jar, jcl-over-slf4j-1.7.5.jar, jackson-core-2.4.4.jar, jackson-databind-2.4.4.jar, jstl-impl-1.2.jar, commons-codec-1.6.jar, httpclient-4.3.6.jar, httpcore-4.3.3.jar, aopalliance-1.0.jar, commons-logging-1.1.3.jar, spring-beans-4.1.7.RELEASE.jar, spring-context-4.1.7.RELEASE.jar, jstl-api-1.2.jar, spring-aop-4.1.7.RELEASE.jar, slf4j-log4j12-1.7.5.jar, spring-expression-4.1.7.RELEASE.jar, standard-1.1.2.jar]}]}, jvmInformation=com.navercorp.pinpoint.profiler.JvmInformation@20403f57}
2020-03-27 01:36:16 [INFO ](c.n.p.p.AgentInfoSender            ) AgentInfo sent.
2020-03-27 01:36:16 [INFO ](c.n.p.p.AgentInfoSender            ) Sending AgentInfo AgentInfo{agentInformation=DefaultAgentInformation{agentId='app-in-docker', applicationName='quickapp', isContainer=false, startTime=1585272660934, pid=1, machineName='563482ed4875', hostIp='172.24.0.12', serverType=TOMCAT, jvmVersion='1.8.0_171', agentVersion='2.0.1'}, serverMetaData=DefaultServerMetaData{serverInfo='', vmArgs=[-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties, -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, -javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar, -Dpinpoint.agentId=app-in-docker, -Dpinpoint.applicationName=quickapp, -Dpinpoint.profiler.profiles.active=release, -Djdk.tls.ephemeralDHKeySize=2048, -Djava.protocol.handler.pkgs=org.apache.catalina.webresources, -Dorg.apache.catalina.security.SecurityListener.UMASK=0027, -Dignore.endorsed.dirs=, -Dcatalina.base=/usr/local/tomcat, -Dcatalina.home=/usr/local/tomcat, -Djava.io.tmpdir=/usr/local/tomcat/temp], connectors={8080=HTTP/1.1, 8009=AJP/1.3}, serviceInfo=[DefaultServiceInfo{serviceName='Catalina/localhost/', serviceLibs=[spring-core-4.1.7.RELEASE.jar, spring-web-4.1.7.RELEASE.jar, slf4j-api-1.7.5.jar, log4j-1.2.17.jar, commons-io-2.4.jar, jackson-annotations-2.4.0.jar, spring-webmvc-4.1.7.RELEASE.jar, jcl-over-slf4j-1.7.5.jar, jackson-core-2.4.4.jar, jackson-databind-2.4.4.jar, jstl-impl-1.2.jar, commons-codec-1.6.jar, httpclient-4.3.6.jar, httpcore-4.3.3.jar, aopalliance-1.0.jar, commons-logging-1.1.3.jar, spring-beans-4.1.7.RELEASE.jar, spring-context-4.1.7.RELEASE.jar, jstl-api-1.2.jar, spring-aop-4.1.7.RELEASE.jar, slf4j-log4j12-1.7.5.jar, spring-expression-4.1.7.RELEASE.jar, standard-1.1.2.jar]}]}, jvmInformation=com.navercorp.pinpoint.profiler.JvmInformation@20403f57}
2020-03-27 01:36:16 [INFO ](c.n.p.p.AgentInfoSender            ) AgentInfo sent.
2020-03-27 01:36:17 [INFO ](c.n.p.p.AgentInfoSender            ) Sending AgentInfo AgentInfo{agentInformation=DefaultAgentInformation{agentId='app-in-docker', applicationName='quickapp', isContainer=false, startTime=1585272660934, pid=1, machineName='563482ed4875', hostIp='172.24.0.12', serverType=TOMCAT, jvmVersion='1.8.0_171', agentVersion='2.0.1'}, serverMetaData=DefaultServerMetaData{serverInfo='', vmArgs=[-Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties, -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager, -javaagent:/pinpoint-agent/pinpoint-bootstrap-2.0.1.jar, -Dpinpoint.agentId=app-in-docker, -Dpinpoint.applicationName=quickapp, -Dpinpoint.profiler.profiles.active=release, -Djdk.tls.ephemeralDHKeySize=2048, -Djava.protocol.handler.pkgs=org.apache.catalina.webresources, -Dorg.apache.catalina.security.SecurityListener.UMASK=0027, -Dignore.endorsed.dirs=, -Dcatalina.base=/usr/local/tomcat, -Dcatalina.home=/usr/local/tomcat, -Djava.io.tmpdir=/usr/local/tomcat/temp], connectors={8080=HTTP/1.1, 8009=AJP/1.3}, serviceInfo=[DefaultServiceInfo{serviceName='Catalina/localhost/', serviceLibs=[spring-core-4.1.7.RELEASE.jar, spring-web-4.1.7.RELEASE.jar, slf4j-api-1.7.5.jar, log4j-1.2.17.jar, commons-io-2.4.jar, jackson-annotations-2.4.0.jar, spring-webmvc-4.1.7.RELEASE.jar, jcl-over-slf4j-1.7.5.jar, jackson-core-2.4.4.jar, jackson-databind-2.4.4.jar, jstl-impl-1.2.jar, commons-codec-1.6.jar, httpclient-4.3.6.jar, httpcore-4.3.3.jar, aopalliance-1.0.jar, commons-logging-1.1.3.jar, spring-beans-4.1.7.RELEASE.jar, spring-context-4.1.7.RELEASE.jar, jstl-api-1.2.jar, spring-aop-4.1.7.RELEASE.jar, slf4j-log4j12-1.7.5.jar, spring-expression-4.1.7.RELEASE.jar, standard-1.1.2.jar]}]}, jvmInformation=com.navercorp.pinpoint.profiler.JvmInformation@20403f57}
2020-03-27 01:36:17 [INFO ](c.n.p.p.AgentInfoSender            ) AgentInfo sent.
`

Sorry to bother you with this, I've had this problem for a long time and I can't solve it, I would be grateful for any help you can give me.

RoySRose commented 4 years ago

Well, I've checked the logs you've posted.

I can see there is an DEADLINE_EXCEEDED exception with grpc, in application log I also can see connection refuse in the collector log.

image

So, I still think there is a problem with the network. If not I also don't know where to start looking at.

Jcmouy commented 4 years ago

Following your guide I was looking at the connection between the servers, I didn't manage to perform the networktest.sh test for GRPC, since apparently in NetworkAvailabilityCheker it only tests the Thrift ports. What I saw that I found interesting is that the agent is sending a lot of information and does not give the time to insert everything and the connection is cut off. From the collector side insert for example until the event 28 and from the agent side the event 29 jump by exception_deadline, from this increase the keepalive values in pinpoint-collector-grpc.properties, and increase also the threadSize values to 128 and queuesize to 1024.

After doing this from the agent side, it started to throw a 3 seconds timeout since it does AgentInfoSender and that throws "request timeout while waiting resoponse".

Digging a little bit more about this I saw an earlier issue that mentioned it and apparently it is linked to the hbase. It was this issue: https://github.com/naver/pinpoint/issues/4119

Searching the hbase log I found the following: 

2020-03-24 07:19:00,876 INFO  [PostOpenDeployTasks:5d9251c4bd257d3f0c5a6e91bc6f330f] regionserver.HRegionServer: Post open deploy tasks for HostApplicationMap_Ver2,\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1585034339670.5d9251c4bd257d3f0c5a6e91bc6f330f.
2020-03-24 07:19:00,879 INFO  [PostOpenDeployTasks:5d9251c4bd257d3f0c5a6e91bc6f330f] hbase.MetaTableAccessor: Updated row HostApplicationMap_Ver2,\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1585034339670.5d9251c4bd257d3f0c5a6e91bc6f330f. with server=33842653d6db,45510,1585034176119
2020-03-24 07:19:00,881 INFO  [AM.ZK.Worker-pool2-t2] master.RegionStates: Transition {5d9251c4bd257d3f0c5a6e91bc6f330f state=OPENING, ts=1585034340820, server=33842653d6db,45510,1585034176119} to {5d9251c4bd257d3f0c5a6e91bc6f330f state=OPEN, ts=1585034340881, server=33842653d6db,45510,1585034176119}
2020-03-24 07:19:00,896 INFO  [ProcedureExecutor-3] zookeeper.ZKTableStateManager: Moving table HostApplicationMap_Ver2 state from ENABLING to ENABLED
2020-03-24 07:19:02,981 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1710b65cbe20006, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2020-03-24 07:19:03,024 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:42420 which had sessionid 0x1710b65cbe20006
Tue Mar 24 07:19:03 UTC 2020 Stopping hbase (via master)

Specifically these lines:

2020-03-24 07:19:02,981 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid 0x1710b65cbe20006, likely client has closed socket
        at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
        at java.lang.Thread.run(Thread.java:745)
2020-03-24 07:19:03,024 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:42420 which had sessionid 0x1710b65cbe20006
Tue Mar 24 07:19:03 UTC 2020 Stopping hbase (via master)

Searching the internet I found the following post: https://issues.apache.org/jira/browse/ZOOKEEPER-1582

Here it details what is a problem with zookeeper that was solved with version 3.4.5. Which makes me wonder which version is using pinpoint-docker? Can you update it? If not, what alternative do I have?

I apologize if I am wrong about any of this, and I thank you for your assistance.

RoySRose commented 4 years ago

Hello, @Jcmouy

Great! First of all, Thank you for reporting about NetworkAvailabilityCheker. I'll look into it.

With the link you've posted(https://issues.apache.org/jira/browse/ZOOKEEPER-1582) the replies from others say that it still didn't work in versions after 3.4.5.

In the docker image in Pinpoint uses v3.4 as shown here https://github.com/naver/pinpoint-docker/blob/master/docker-compose.yml https://github.com/naver/pinpoint-docker/blob/master/pinpoint-zookeeper/docker-compose.yml

So, here is my suggestion. Why don't you try updating the version of pinpoint-zookeepers to 3.4.5 or 3.5 or 3.6, since it's available https://hub.docker.com/_/zookeeper (and it would take only several minutes. since it's the DOCKER IMAGE) And then you can make a pull request and become one of the contributors!

Jcmouy commented 4 years ago

Hi Roy, I made the corresponding change in the yml file, so that it brings me the 3.5 version of the zookeeper image, but I encountered the following problem. In the collector's log it shows me the following:

2020-03-30 23:01:51 [WARN ](c.n.p.c.c.z.ZookeeperJobWorker     ) handleUpdate failed. caused:Instance must be connected., jobSize:1
com.navercorp.pinpoint.common.server.cluster.zookeeper.exception.ConnectionException: Instance must be connected.
        at com.navercorp.pinpoint.common.server.cluster.zookeeper.CuratorZookeeperClient.checkState(CuratorZookeeperClient.java:239)
        at com.navercorp.pinpoint.common.server.cluster.zookeeper.CuratorZookeeperClient.createPath(CuratorZookeeperClient.java:76)
        at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.handleUpdate(ZookeeperJobWorker.java:318)
        at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.handle(ZookeeperJobWorker.java:300)
        at com.navercorp.pinpoint.collector.cluster.zookeeper.ZookeeperJobWorker.run(ZookeeperJobWorker.java:205)
        at java.lang.Thread.run(Thread.java:748)

Looking around I saw that this was a bug linked to the hbase and zookeeper, so I decided to look at the hbase log and found the following:

020-03-30 22:43:09,313 ERROR [main] master.HMasterCommandLine: Master exiting
java.io.IOException: Could not start ZK at requested port of 2181.  ZK was started at port: 2182.  Aborting as clients (e.g. shell) will not be able to find this ZK quorum.
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:212)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:137)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2522)

Apparently this problem is linked to both hbase and zookeeper wanting to start on the same port, which is why it fails. More information can be seen at this link: https://stackoverflow.com/questions/17638626/hbase-master-not-able-to-start/48660423.

Is there a way to fix it?

Thank you very much!

RoySRose commented 4 years ago

Hello, @Jcmouy

if you only changed the versions in the corresponding yml file, why is there any port conflict problem? since that yml was used for a while. And as you can see, ports 2888:3888 is used for the zookeepers

image

Jcmouy commented 4 years ago

Hi Roy, but unless I'm mistaken the problem is on port 2181 and not the one you're pointing at. I'm taking this off this line:

java.io.IOException: Could not start ZK at requested port of 2181. ZK was started at port: 2182. Aborting as clients (e.g. shell) will not be able to find this ZK quorum.

What I could understand from what I read, is that both hbase and zookeeper want to start on the same port causing a conflict.

Am I wrong?

RoySRose commented 4 years ago

Do you mean there isn't any confliction problem if you use zookeeper 3.4 and have one when you use 3.5?