FinalizableReferenceQueue still leaks

[gissuebot]

From gili.tzabari on December 25, 2008 12:51:41

Issue 227 is closed but the problem was not really fixed.

Stuart's comment #8 explains what remains to be fixed: https://code.google.com/p/google-guice/issues/detail?id=227&can=1&start=200#c8

Original issue: http://code.google.com/p/google-guice/issues/detail?id=288

[gissuebot]

From jose.illescas on January 28, 2011 07:31:30

Let me, explain...

Initialize Executor on injector instance

  Injector injector = Guice.createInjector(myModule, myExecutor);

You can initialize Executor on Injector constructor => new Injector(myExecutor) or init method after constructor call => new Injector().init(myExecutor)

Destroying, with non static method (using "previous" instance of injector)

  injector.destroy();

[gissuebot]

From tj.rothwell on January 28, 2011 08:50:03

How early do we need to have the instance of an executor?

Thread started here: https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/Finalizer.java?r=830#82 Which is only invoked by: https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/FinalizableReferenceQueue.java?r=830#117 Which is only invoked by: https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/MapMaker.java?r=830#773 Which is only invoked when the private static class MapMaker.QueueHolder is accessed at these locations: 1) https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/MapMaker.java?r=830#873 2) https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/MapMaker.java?r=830#931 3) https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/MapMaker.java?r=830#989 4) https://code.google.com/p/google-guice/source/browse/trunk/src/com/google/inject/internal/MapMaker.java?r=830#1014 Which would occur when MapMaker creates  soft/weak keys or soft/weak references. (Does it create any of those before entries are added?)

We need the executor when we obtain the ReferenceQueue for FinalizerReferenceQueue (frq).

We should have the ability to access the executor instance before MapMaker creation. We must have the ability to access the executor instance before MapMaker addition. (unless I'm mistaken)

For a Guice application, when is the first MapMaker instance created as well as the first addition to it? I'm running out of time for this research, so I need help here.

[gissuebot]

From fry@google.com on January 28, 2011 08:53:39

FYI, I'll be submitting a change to MapMaker (hopefully today) which will simply remove its dependence on FinalizableReferenceQueue.

[gissuebot]

From tj.rothwell on January 28, 2011 08:56:01

BTW, one more thing, the QueueHolder static init is what limits MapMaker to a single frq instance.

[gissuebot]

From sameb@google.com on January 28, 2011 09:00:24

After we validate fry's change is stable (and after I get internet at home), I'll be updating the embedded guava code in guice.

[gissuebot]

From fry@google.com on January 28, 2011 09:00:57

Well, soon enough there will be zero frq instances.

[gissuebot]

From nimbus4321 on February 08, 2011 10:27:31

Hi Fry,

Does your fix also address issue 488 ?

[gissuebot]

From fry@google.com on February 08, 2011 10:54:00

yes.

[gissuebot]

From sberlin on February 21, 2011 17:52:15

(No comment was entered for this change.)

Labels: -Milestone-Release2.0

[gissuebot]

From larrrs.jung on March 05, 2011 18:51:49

Hi, is there any fix available? I'm using Guice 3.0 rc2 (Jan 09) and have the same issue.

I get the following messages:

1x The web application [/xxx] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.

mutiple times: The web application [/xxx] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@4604a96a]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@40974600]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

[gissuebot]

From fry@google.com on March 05, 2011 19:55:37

No fix yet. This issue will be updated when there is. :-)

[gissuebot]

From larrrs.jung on March 05, 2011 19:58:54

okay, thx for the reply :)

[gissuebot]

From nimbus4321 on March 07, 2011 14:07:10

Hi Fry, I couldn't see the fix in the recent Guice 3 RC3 release - is it still going to make it into the GA release ? thanks.

[gissuebot]

From fry@google.com on March 07, 2011 14:10:51

There is no fix yet. This issue will be updated when there is. :-/

[gissuebot]

From sberlin on March 07, 2011 14:30:03

It is unlikely the fix will be in Guice 3.0.  We'll update the Guice source with the latest Guava bits when it's available.

[gissuebot]

From jennifer.anagrius on March 28, 2011 06:58:55

Are there any quick and dirty workarounds for this atm? I'm really in a bind here :(

[gissuebot]

From mcculls on March 28, 2011 09:51:21

One quick and dirty workaround would be to use the sisu-guice binary: http://repo1.maven.org/maven2/org/sonatype/sisu/sisu-guice/3.0.0/ This is guice 3.0 plus some patches ( https://github.com/sonatype/sisu-guice/blob/master/PATCHES ).

One of these changes is to add a property "-Dguice.executor.class=Clazz" where Clazz implements java.util.concurrent.Executor. You can then supply your own executor to manage the finalizer work. You can also use "-Dguice.executor.class=NONE" to turn off the finalizer completely, but note that this means cleanup will only happen occasionally when the weak collections mutate.

HTH  (sorry about the use of a property, I didn't want to introduce an experimental API that wasn't in Guice)

[gissuebot]

From bontecyril on March 31, 2011 04:56:28

As another workaround that doesn't require to patch guice sources, I've started to work on a Tomcat Listener that try to shutdown the thread when the context is destroyed. It also tries to cancel the Expiration Timer launched by MapMaker.

It looks to work correctly (with both guice 2.0 and 3.0) but I've not tested it in production yet. The code can easily be ported to other applications thant tomcat.

The Listener sources are in attachment (note that I removed the package name from the sources). Maybe it can be a starting point to be integrated in guice or tomcat, or in a "guice-utils" jar.

Attachment: gist    GuiceLifecycleListener.java

[gissuebot]

From jose.illescas on May 09, 2011 02:37:09

@bontecy, I test your GuiceLifecycleListener on my Tomcat (development environment) without any Permgen errors on redeploys... GREAT!!

[gissuebot]

From bontecyril on May 09, 2011 05:59:05

Ok, great. Note that I found a conflict with this workaround when a second context provides the jars but don't use them : the listener then initializes the thread for that context while it tries to stop the thread of the first context. Also, the loops are not optimal. I didn't find time to fix this for now but I'll try to address this if no solution comes soon in guice (Then, I'll probably create a repository outside of this bug report).

[gissuebot]

From fry@google.com on June 13, 2011 07:11:26

The fix for this has finally landed in Guava, and will be deployed in release 10.

[gissuebot]

From henrychankinwo on July 12, 2011 18:09:04

Will there be a new Guice release containing this fix?

[gissuebot]

From jose.illescas on September 29, 2011 04:14:39

Now guava (10) fixed this issue. Any plan/roadmap to include this fix on guice?

[gissuebot]

From colin.taylor on October 02, 2011 18:12:38

Any word on a fix? Nothing mentioned so far works for me..

[gissuebot]

From mcculls on October 02, 2011 18:36:18

You could always try sisu-guice 3.1.0 (which is guice + a few experimental patches, such as the guava update) http://search.maven.org/#artifactdetails%7Corg.sonatype.sisu%7Csisu-guice%7C3.1.0%7Cjar Note that this depends on sisu-guava 0.9.9 - which was based on pre-10 build of guava with the finalizer thread fix, but you can always substitute this with guava 10 now that's been released. Also unlike previous releases this particular flavour of guice doesn't embed/jarjar guava code, so you do need both on the classpath.

[gissuebot]

From mcculls on October 07, 2011 03:49:10

Minor patch to update Guava dependency to 10.0

Attachment: gist    _update_to_guava_10.patch_ Binary attachments: guava-10.0.jar

[gissuebot]

From jose.illescas on October 26, 2011 04:03:12

Great work mcculls,

 But i like some plan/roadmap to include this fix on "original" guice?

For example: a "official" guice 3.1 release to resolve all Accepted issues with Hight priority must be one ideal solution...

[gissuebot]

From mcculls on November 10, 2011 11:25:10

Issue 630 has been merged into this issue.

[gissuebot]

From jose.illescas on November 11, 2011 09:20:48

@mcculls,

I try your patch with guava-10.0 but tomcat persist on:

[16] SEVE 18:16:32 WebappClassLoader.checkThreadLocalMapForLeaks: The web application [/ulysses] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@125ec471]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@bb2e023]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak. [16] SEVE 18:16:32 WebappClassLoader.checkThreadLocalMapForLeaks: The web application [/ulysses] created a ThreadLocal with key of type [com.google.inject.internal.InjectorImpl$1] (value [com.google.inject.internal.InjectorImpl$1@125ec471]) and a value of type [java.lang.Object[]] (value [[Ljava.lang.Object;@c991fd5]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Any fix/idea?

[gissuebot]

From stefan.simroth on November 11, 2011 09:27:37

Hi, I resolved the issue for me by compiling sisu-guice (master) with the guava 10.0.1 dependency (instead of the sisu-guava 0.9.9). See here: https://github.com/sas101/sisu-guice/commit/2bc6e1554d29faf95df3cb959d258456ceaf060a Thx @mcculls for providing that suggestion.

[gissuebot]

From mcculls on November 11, 2011 09:34:20

As explained above the remaining ThreadLocal of type Object[1] seen by Tomcat is not a memory leak and would eventually get cleared. However, even if it wasn't cleared it only takes up a few bytes and would not hold onto any user level classloaders, which is the cause of most leaks. This is because the single element in that array is guaranteed to be null'd out in a finally clause and the component type of the array is Object.

Note: the reason the ThreadLocal is left between injector calls and not removed on every call is for performance reasons. Explicitly removing a ThreadLocal is not mandated as shown in the example from the official javadocs http://download.oracle.com/javase/6/docs/api/java/lang/ThreadLocal.html because it should automatically get cleared when the ThreadLocal is no longer accessible. Unfortunately the JDK is slow in clearing out stale entries, which coupled with the (overly) zealous checks in Tomcat can lead to these spurious error messages (which should really be warnings imho).

[gissuebot]

From balint@krivan.info on July 12, 2012 00:10:51

Using Guice 3.0 from central maven repo I am still getting this:

2012.07.12. 9:00:18 org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [/foo-bar-1.0-SNAPSHOT] appears to have started a thread named [com.google.inject.internal.util.$Finalizer] but has failed to stop it. This is very likely to create a memory leak.

Any news on this? I don't see guava dependency in the pom, but I've read it will be fixed once it is upgraded. Thanks!

[gissuebot]

From mcculls on July 20, 2012 11:11:56

Note that Guice 3.0 still embeds an old version of Guava (which is why the pom has no explicit dependency to it) so that message is expected.

[gissuebot]

From pas256 on July 20, 2012 11:17:52

OMG, we are getting close to 4 years old for this one... way to go Google!

[gissuebot]

From mcculls on July 20, 2012 11:23:14

PS. you you can always use sisu-guice (which doesn't embed guava, but instead has it as a dependency) http://search.maven.org/#artifactdetails%7Corg.sonatype.sisu%7Csisu-guice%7C3.1.1%7Cjar Or you could compile from source - also note in many scenarios the finalizer thread is not actually an issue.

[gissuebot]

From s.a.grigoriev on July 28, 2012 07:51:18

This definitely needs to be fixed.

[gissuebot]

From mcculls on August 06, 2012 18:25:21

Here's an additional patch that clears up the last of the Tomcat warnings after upgrading to the latest Guava (the ones referring to the injectorImpl's ThreadLocal). The issue here was the injector used an anonymous class to define a custom initialValue method for its localContext ThreadLocal. Unfortunately this anonymous class has an implicit reference (seen as this$0 in the bytecode) back to the instance where it was created, ie. the injector. We could change this to be a static nested class which would avoid the reference to the injector, but this still keeps a reference back to the containing classloader which upsets Tomcat's strict checks. This patch removes the custom ThreadLocal subclass and adds a check+set in the callInContext method - note this is slightly less performant, since it needs an extra call to set(). It's questionable whether this performance hit is worth getting rid of all the Tomcat warnings, switching to a static nested class may be enough since you'd still occasionally see a warning but the only thing that a rogue thread local might be holding onto would be the classloader. Anyway, here's the patch in case anyone is interested in trying it out with trunk...

Attachment: gist    _GUICE_288_decouple_thread_local.patch_

[gissuebot]

From mcculls on August 07, 2012 12:00:50

Issue 698 has been merged into this issue.

[gissuebot]

From nicolas.antoniazzi on August 24, 2012 02:01:51

I tried the last version from trunk + the patch from Stuart McCulloch, and it works now perfectly. This patch should be integrated to guice.

Thanks

[gissuebot]

From mcculls on September 02, 2012 03:32:55

Issue 698 has been merged into this issue.

[gissuebot]

From lorenzom on January 21, 2013 07:57:37

If the proposed decoupling thread local patch is the fix we need and want, then wouldn't it be more conservative to declare a static concrete inner class for the LocalContext ? We would not need to change the lifecycle of InjectorImpl.localContext.

Attachment: gist    _GUICE_288_static_concrete_class.patch_

[gissuebot]

From mcculls on February 27, 2013 15:12:40

As mentioned in #c88 using a static nested class would still leave an indirect reference to the guice classloader which would be picked up by Tomcat's memory leak detector. So if you wanted to remove all warnings then the patch in #c88 is the best I've found so far.

[gissuebot]

From ryan@asleson.net on October 16, 2013 05:35:18

So this bug has not been fixed in the Guice 4.0 beta, correct?  So is it correct to assume that it won't make it into 4.0 GA?  Bummer.

[gissuebot]

From mcculls on October 16, 2013 06:00:36

Guice 4.0-beta embeds Guava 11.0.1, which includes the FinalizableReferenceQueue fix. You may still see a warning in Tomcat about a ThreadLocal (see comment #c88 and associated patch) but this is relatively minor compared to the original issue.

[gissuebot]

From cowwoc@bbs.darktech.org on October 16, 2013 08:10:20

All of this could have been solved 6 (!!) years ago by simply adding a close() method. See comment #3.

Add a close() method now and remove it in the future if you ever get an alternative working.

[gissuebot]

From mcculls on October 16, 2013 08:21:22

There's no need for a close() method now - the weak/soft maps were re-implemented in Guava 10 to avoid the need for the FRQ background thread, so there is literally nothing to close: https://code.google.com/p/guava-libraries/source/detail?r=a13e02167e90125e6a78bf9bbd061996d05a143a . The FRQ fix is available in Guice 4.0-beta for testing.

[gissuebot]

From mcculls on December 01, 2013 17:23:50

Issue 786 has been merged into this issue.

[gissuebot]

From sberlin on December 05, 2013 15:51:07

Issue 707 has been merged into this issue.

[gissuebot]

From mvdgraaf on March 01, 2014 06:12:14

I see that there have been multiple attempts to fix the localContext variable in the InjectorImpl. But sadly they all fail to solve the issue completely and specially in a Servlet container.

In tomcat there is still a warning (also in 4.0-beta) because the ThreadLocal is not cleared (the object[] is still leaked into the Threads).

With the current JVM's it requires a new ThreadLocal to do something to finally garbage collect  ThreadLocal's values from the Thread class and each thread needs to get hit with that new ThreadLocal to completely cleanup the mess left behind by other ThreadLocal's this can take some time for an application server like tomcat because it can run 100+ Thread's and there is no guarantee that the Thread will ever hit a new ThreadLocal.

To solve these issues we need to clear the ThreadLocal and must not use a library/application extends class to initialize the ThreadLocal.

The attached patch file removes the LocalContextThreadLocal, removes the unnecessary Object[] and clears the ThreadLocal.

Attachment: gist    ThreadLocalFix-guice.patch

[gissuebot]

From mcculls on March 01, 2014 06:18:53

IIRC the use of the holder array was for performance reasons, but there is a way to keep the holder array and still avoid the implicit classloader reference and associated Tomcat warning: https://github.com/sonatype/sisu-guice/blob/master/PATCHES/GUICE_288_decouple_thread_local.patch