Too many open files - related to dtags.eftar (Bugzilla #19178)

[vladak]

status ACCEPTED severity normal in component indexer for next Reported in version unspecified on platform ANY/Generic Assigned to: Trond Norbye

On 2012-02-17 06:14:11 +0000, Ragesh Nair wrote:

We have upgrade our OpenGrok setup to latest 0.11 release and we found that there are too many open files left open while running Opengrok.

We have Ubuntu 64 bit 11.10 oneiric where OpenGrok is deployed in Apache 7.

Its all same as the one we filed earlier https://defect.opensolaris.org/bz/show_bug.cgi?id=17127 but "data/index/dtags.eftar (deleted)" is what getting accumulated this time.

lsof|grep dtags | wc -l

4448

java 30882 tomcat7 42r REG 8,17 32 95472116 /OUR/data/index/dtags.eftar (deleted) java 30882 tomcat7 43r REG 8,17 32 95472116 /OUR/data/index/dtags.eftar (deleted) java 30882 tomcat7 44r REG 8,17 32 95472116 /OUR/data/index/dtags.eftar (deleted)

Above are seen in many numbers.

Please do the needful and let me know if you need any more information in this regard.

On 2012-02-17 09:02:41 +0000, Lubos Kosco wrote:

we have a mitigation for this in http://defect.opensolaris.org/bz/show_bug.cgi?id=19169

however it seems in 0.11 we don't get web/pageconfig.jspf# 33 executed properly

On 2012-02-17 09:30:18 +0000, Lubos Kosco wrote:

a quick remedy might be to move /OUR/data/index/dtags.eftar aside until we release a fix ...

On 2012-04-16 00:23:48 +0000, Jens Elkner wrote:

see http://defect.opensolaris.org/bz/show_bug.cgi?id=19194

[vladak]

The link to Bugzilla bug 19194 is now issue #543.

[vladak]

With the fix for #539 I am marking this as stopper since it made this visible.

[vladak]

The patch 1367.patch from #543 which should fix this problem:

# HG changeset patch
# User jel+opengrok@cs.uni-magdeburg.de
# Date 1331531570 -3600
# Node ID cdb5b6d69abf84cebd61df01aaee236f449aec24
# Parent  57e41a6d7dbed2b8c7f8f6b7d5ddcc904be6bc5d
#19178: fix eftar fd leak

diff -r 57e41a6d7dbe -r cdb5b6d69abf src/org/opensolaris/opengrok/web/PageConfig.java
--- a/src/org/opensolaris/opengrok/web/PageConfig.java  Mon Mar 12 05:54:30 2012 +0100
+++ b/src/org/opensolaris/opengrok/web/PageConfig.java  Mon Mar 12 06:52:50 2012 +0100
@@ -43,6 +43,7 @@
 import java.util.logging.Logger;
 import java.util.regex.Pattern;

+import javax.servlet.ServletRequest;
 import javax.servlet.http.Cookie;
 import javax.servlet.http.HttpServletRequest;

@@ -61,9 +62,9 @@

 /**
  * A simple container to lazy initialize common vars wrt. a single request.
- * It MUST NOT be shared between several requests and {@link #cleanup()} should
- * be called before the page context gets destroyed (e.g. by overwriting
- * {@code jspDestroy()} or when leaving the {@code service} method.
+ * It MUST NOT be shared between several requests and
+ * {@link #cleanup(ServletRequest)} should be called before the page context 
+ * gets destroyed (e.g.when leaving the {@code service} method).
  * <p>
  * Purpose is to decouple implementation details from web design, so that the
  * JSP developer does not need to know every implementation detail and normally
@@ -1153,6 +1154,7 @@
         request.setAttribute(ATTR_NAME, pcfg);
         return pcfg;
     }
+    
     private static final String ATTR_NAME = PageConfig.class.getCanonicalName();
     private HttpServletRequest req;

@@ -1161,17 +1163,24 @@
     }

     /**
-     * Cleanup all allocated resources. Should always be called right before
-     * leaving the _jspService / service.
+     * Cleanup all allocated resources (if any) from the instance attached to 
+     * the given request.
+     * @param sr request to check, cleanup. Ignored if {@code null}.
+     * @see PageConfig#get(HttpServletRequest)
+     * 
      */
-    public void cleanup() {
-        if (req != null) {
-            req.removeAttribute(ATTR_NAME);
-            req = null;
+    public static void cleanup(ServletRequest sr) {
+        if (sr == null) {
+            return;
         }
-        env = null;
-        if (eftarReader != null) {
-            eftarReader.close();
+        PageConfig cfg = (PageConfig) sr.getAttribute(ATTR_NAME);
+        if (cfg == null) {
+            return;
+        }        
+        sr.removeAttribute(ATTR_NAME);
+        cfg.env = null;
+        if (cfg.eftarReader != null) {
+            cfg.eftarReader.close();
         }
     }
 }
diff -r 57e41a6d7dbe -r cdb5b6d69abf src/org/opensolaris/opengrok/web/WebappListener.java
--- a/src/org/opensolaris/opengrok/web/WebappListener.java  Mon Mar 12 05:54:30 2012 +0100
+++ b/src/org/opensolaris/opengrok/web/WebappListener.java  Mon Mar 12 06:52:50 2012 +0100
@@ -35,6 +35,8 @@
 import javax.servlet.ServletContext;
 import javax.servlet.ServletContextEvent;
 import javax.servlet.ServletContextListener;
+import javax.servlet.ServletRequestEvent;
+import javax.servlet.ServletRequestListener;

 import org.opensolaris.opengrok.configuration.RuntimeEnvironment;

@@ -42,7 +44,9 @@
  * Populate the Mercurial Repositories
  * @author Trond Norbye
  */
-public final class WebappListener  implements ServletContextListener {
+public final class WebappListener 
+    implements ServletContextListener, ServletRequestListener
+{
     /** Boolean servlet parameter to indicate, whether to serve debug enabled JS */ 
     public static final String JS_DEBUG_PARAM = "JsDebug";

@@ -125,4 +129,20 @@
             ctx.setAttribute(name, Boolean.FALSE);
         }
     }
+
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public void requestDestroyed(ServletRequestEvent e) {
+        PageConfig.cleanup(e.getServletRequest());
+    }
+
+    /**
+     * {@inheritDoc}
+     */
+    @Override
+    public void requestInitialized(ServletRequestEvent e) {
+        // pass through
+    }
 }
diff -r 57e41a6d7dbe -r cdb5b6d69abf web/pageconfig.jspf
--- a/web/pageconfig.jspf   Mon Mar 12 05:54:30 2012 +0100
+++ b/web/pageconfig.jspf   Mon Mar 12 06:52:50 2012 +0100
@@ -26,11 +26,4 @@
 org.opensolaris.opengrok.web.PageConfig"
 %><%!
     private PageConfig cfg;
-    /** Just cleanup cached objects */
-
-    public void jspDestroy() {
-        if (cfg != null) {
-            cfg.cleanup();
-        }
-    }
 %>
\ No newline at end of file