[Agent] can not connect to the collector maybe cause memory leak

xuet0ng commented 5 years ago

Please answer these questions before submitting your issue.

Why do you submit this issue?
[ ] Question or discussion
[x] Bug
[ ] Requirement
[ ] Feature or performance improvement

Bug

Which version of SkyWalking, OS and JRE?
- version: v5.0.0-GA
- OS: centos 6.5
- JRE: java version "1.8.0_144"
Which company or project?

What happen? If possible, provide a way for reproducing the error. e.g. demo application, component version.

background
java, spring cloud, micro-service
what happened
several services unavailable on different machine and almost in same time.

java service log says

java.lang.OutOfMemoryError: GC overhead limit exceeded

skywalking log says

00:02:21.725 [SkywalkingAgent-2-GRPCChannelManager-0] ERROR o.a.s.a.d.i.g.i.ManagedChannelOrphanWrapper - *~*~*~ Channel ManagedChannelImpl{logId=1042418, target=xxxxx:xxxxx} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.
java.lang.RuntimeException: ManagedChannel allocation site
    at org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelOrphanWrapper$ManagedChannelReference.<init>(ManagedChannelOrphanWrapper.java:103)
    at org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:53)
    at org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelOrphanWrapper.<init>(ManagedChannelOrphanWrapper.java:44)
    at org.apache.skywalking.apm.dependencies.io.grpc.internal.AbstractManagedChannelImplBuilder.build(AbstractManagedChannelImplBuilder.java:410)
    at org.apache.skywalking.apm.agent.core.remote.GRPCChannel.<init>(GRPCChannel.java:46)
    at org.apache.skywalking.apm.agent.core.remote.GRPCChannel.<init>(GRPCChannel.java:31)
    at org.apache.skywalking.apm.agent.core.remote.GRPCChannel$Builder.build(GRPCChannel.java:95)
    at org.apache.skywalking.apm.agent.core.remote.GRPCChannelManager.run(GRPCChannelManager.java:99)
    at org.apache.skywalking.apm.util.RunnableWithExceptionProtection.run(RunnableWithExceptionProtection.java:36)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
00:03:21.724 [SkywalkingAgent-2-GRPCChannelManager-0] ERROR o.a.s.a.d.i.g.i.ManagedChannelOrphanWrapper - *~*~*~ Channel ManagedChannelImpl{logId=1042467, target=xxxxxx:xxxxx} was not shutdown properly!!! ~*~*~*
Make sure to call shutdown()/shutdownNow() and wait until awaitTermination() returns true.

java abnormal objects use 'jmap -histo:live {pid} > ~/histolive'

num     #instances         #bytes  class name
----------------------------------------------
[xxx] [xxx@xxxxxxxxxxxx ~]$ grep '39013' histolive 
4:         39013       20598600  [Lorg.apache.skywalking.apm.dependencies.io.netty.handler.codec.http2.HpackHeaderField;
11:         39013        8426808  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
30:         39013        3745248  org.apache.skywalking.apm.dependencies.io.grpc.internal.TransportTracer
40:         39013        2808936  org.apache.skywalking.apm.dependencies.io.netty.channel.DefaultChannelHandlerContext
47:         39013        2496832  org.apache.skywalking.apm.dependencies.io.grpc.internal.DelayedClientTransport
48:         39013        2496832  org.apache.skywalking.apm.dependencies.io.grpc.netty.NettyChannelBuilder$NettyTransportFactory
57:         39013        1872624  org.apache.skywalking.apm.dependencies.io.grpc.internal.Rescheduler
94:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule
95:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusTracingModule
96:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
97:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
98:         39013        1248416  org.apache.skywalking.apm.dependencies.io.grpc.internal.ServiceConfigInterceptor
99:         39013        1248416  org.apache.skywalking.apm.dependencies.io.netty.channel.AdaptiveRecvByteBufAllocator
128:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.AtomicBackoff
129:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CallCredentialsApplyingTransportFactory
130:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule$1
131:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusStatsModule$StatsClientInterceptor
132:         39013         936312  org.apache.skywalking.apm.dependencies.io.grpc.internal.CensusTracingModule$1

skywalking agent code fragment

package org.apache.skywalking.apm.agent.core.remote;
public class GRPCChannelManager ... {
...
@Override
public void run() {
   logger.debug("Selected collector grpc service running, reconnect:{}.", reconnect);
   if (reconnect) {
       if (RemoteDownstreamConfig.Collector.GRPC_SERVERS.size() > 0) {
           String server = "";
           try {
               int index = Math.abs(random.nextInt()) % RemoteDownstreamConfig.Collector.GRPC_SERVERS.size();
               server = RemoteDownstreamConfig.Collector.GRPC_SERVERS.get(index);
               String[] ipAndPort = server.split(":");

               managedChannel = GRPCChannel.newBuilder(ipAndPort[0], Integer.parseInt(ipAndPort[1]))
                   .addManagedChannelBuilder(new StandardChannelBuilder())
                   .addManagedChannelBuilder(new TLSChannelBuilder())
                   .addChannelDecorator(new AuthenticationDecorator())
                   .build();

               if (!managedChannel.isShutdown() && !managedChannel.isTerminated()) {
                   reconnect = false;
                   notify(GRPCChannelStatus.CONNECTED);
               } else {
                   notify(GRPCChannelStatus.DISCONNECT);
               }
               return;
           } catch (Throwable t) {
               logger.error(t, "Create channel to {} fail.", server);
               notify(GRPCChannelStatus.DISCONNECT);
           }
       }

       logger.debug("Selected collector grpc service is not available. Wait {} seconds to retry", Config.Collector.GRPC_CHANNEL_CHECK_INTERVAL);
   }
}
...
}

if grpc original channel can not connect the collector...yes, the collector of this env being dead more than 10 days... :(
1. original channel will not shutdown
2. a new original channel will be created
3. memory leak
4. until gc can not reclaim memory, out of memory

wu-sheng commented 5 years ago

Could you try that, shutdown the old one before creating a new one. Could that solve this leak?

Look forward your feedback and update.

xuet0ng commented 5 years ago

original jar

start & connect collector success

➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
1252:             1            216  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
2978:             1             40  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
3361:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
3362:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
4125:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
4126:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
4127:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
4128:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
4129:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
5484:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
5485:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
5486:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
5487:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
5488:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent
➜  skywalking-agent
➜  skywalking-agent
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
1246:             1            216  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
2985:             1             40  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
3372:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
3373:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
4130:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
4131:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
4132:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
4133:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
4134:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
5485:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
5486:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
5487:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
5488:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
5489:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel

disconnect network & wait for 3 mins

➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
1246:             1            216  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
2992:             1             40  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
3387:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
3388:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
4152:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
4153:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
4154:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
4155:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
4156:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
5511:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
5512:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
5513:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
5514:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
5515:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
882:             2            432  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
2017:             2             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
2352:             2             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
2353:             2             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
2691:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
2692:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
2693:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
2694:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
2695:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
3407:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
3408:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
3409:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
3410:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
3411:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent
➜  skywalking-agent
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
732:             3            648  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1676:             3            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1876:             3             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1877:             3             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
2159:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
2160:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
2161:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
2162:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
2163:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
2714:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
2715:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
2716:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
2717:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
2718:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
663:             4            864  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1437:             4            160  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1578:             4            128  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1579:             4            128  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
1910:             4             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
1911:             4             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
1912:             4             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
1913:             4             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
1914:             4             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
2408:             4             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
2409:             4             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
2410:             4             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
2411:             4             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
2412:             4             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
601:             5           1080  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1290:             5            200  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1458:             5            160  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1459:             5            160  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
1727:             5            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
1728:             5            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
1729:             5            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
1730:             5            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
1731:             5            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
2090:             5             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
2091:             5             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
2092:             5             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
2093:             5             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
2094:             5             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
559:             6           1296  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1186:             6            240  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1343:             6            192  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1344:             6            192  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
1554:             6            144  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
1555:             6            144  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
1556:             6            144  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
1557:             6            144  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
1558:             6            144  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
1942:             6             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
1943:             6             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
1944:             6             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
1945:             6             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
1946:             6             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 21310 | grep ManagedChannelImpl
524:             7           1512  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1098:             7            280  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1267:             7            224  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1268:             7            224  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
1441:             7            168  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
1442:             7            168  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
1443:             7            168  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
1444:             7            168  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
1445:             7            168  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
1820:             7            112  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
1821:             7            112  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
1822:             7            112  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
1823:             7            112  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
1824:             7            112  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel

modified code if reconnect & old channel shutdownNow

start & connect collector success

➜  skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl
1250:             1            216  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
2999:             1             40  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
3385:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
3386:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
4140:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
4141:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
4142:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
4143:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
4144:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
5495:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
5496:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
5497:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
5498:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
5499:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent
➜  skywalking-agent
➜  skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl
1251:             1            216  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
2977:             1             40  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
3367:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
3368:             1             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
4136:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
4137:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
4138:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
4139:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
4140:             1             24  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
5499:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
5500:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
5501:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
5502:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
5503:             1             16  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel

disconnect network & wait for 3 mins

➜  skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl
732:             3            648  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1675:             3            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1874:             3             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1875:             3             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
2150:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
2151:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
2152:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
2153:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
2154:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
2708:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
2709:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
2710:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
2711:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
2712:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl
878:             2            432  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1999:             2             80  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
2338:             2             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
2339:             2             64  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
2681:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
2682:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
2683:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
2684:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
2685:             2             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
3399:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
3400:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
3401:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
3402:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
3403:             2             32  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel
➜  skywalking-agent jmap -histo:live 20956 | grep ManagedChannelImpl
732:             3            648  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl
1675:             3            120  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$SubchannelImpl
1874:             3             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1
1875:             3             96  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$UncommittedRetriableStreamsRegistry
2150:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$3
2151:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$6
2152:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl
2153:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$LbHelperImpl$1
2154:             3             72  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$NameResolverListenerImpl
2706:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$1AutoDrainChannelExecutor
2707:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$2
2708:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$5
2709:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$IdleModeTimer
2710:             3             48  org.apache.skywalking.apm.dependencies.io.grpc.internal.ManagedChannelImpl$RealChannel

中文了...这是本地电脑上的验证结果..
1. 不修改...一直能链接上collector..链接3分钟...ManagedChannelImpl的对象数一直是1
2. 不修改...先链接collector..再断网..断网3分钟...ManagedChannelImpl的对象数7...中途几次查看对象数目一直在飙升
3. 修改...先链接collector..再断网..断网3分钟...ManagedChannelImpl的对象数3...中途几次查看对象数目1,2,3,2,3...有shutdown关闭释放的迹象，猜测但可能存在shutdown慢或者shutdown失败...

xuet0ng commented 5 years ago

修改方法是...reconnect的时候，将原来channel.shutdownNow()

wu-sheng commented 5 years ago

@xuet0ng Do you want to raise a pull request to fix this? You are the one found this and found the solution. You should take the credit.

xuet0ng commented 5 years ago

@wu-sheng Thanks for the invitation, but unfortunately, sorry and I am afraid I can't do this. Not familiar with Skywalking ,the first time I began being familiar with it was the day before yesterday downloaded the code to locate the memory overflow problem . And the project in my company is too tight now, can't guarantee my energy to complete this thing.

wu-sheng commented 5 years ago

Sure. I could submit a pr to fix this. Thank you to find the issue and recheck the possible fix.

xuet0ng commented 5 years ago

by the way, tried and tested shutdownNow old channel if reconnect... 1st, start & connect the collector normally 2nd, disconnect the network and wait for 10 mins 3rd, reconnect the network, reconnect collector again and wait for 10 mins

in 2nd leaked instances increased very slowly in 3rd instances reduced to 2, I assumed that "to 1", interesting...

it works, but can not solve all the problems, but maybe it is acceptable.

wu-sheng commented 5 years ago

There may be still some uncertain scenario. I will try to improve that codes. And for agent-backend, I prefer to put a load balance between then, and payload of collector should be better, and we could leave the reconnect to grpc itself.

wu-sheng commented 5 years ago

Fixed.

apache / skywalking

[Agent] can not connect to the collector maybe cause memory leak #2190

Bug