[JENKINS-41100] Memory leak in web interface (Javascript)

[timja]

Tested version: jenkins 2.39
Browser: Google Chrome

Issue:

If I open a browser tab on Jenkins and let it open, the memory consumed by the tab grows up to 2GB (and then I need to close it to be able to work). Consumed memory can be accessed by typing `Shift + Esc` in Google Chrome.

When I just open the home view, it starts around 52 kB

I suppose there is a memory leak in the JS code.

If I launch the Memory profiler in Chrome, it appears that objects are created every 5s and they're never released (See attachment).

The memory leak seems related to event handling (see 2nd attachment)

---

Originally reported by gga, imported from: Memory leak in web interface (Javascript)

• status: Open
• priority: Major
• resolution: Unresolved
• imported: 2022/01/10

[timja]

danielbeck:

Weird. JENKINS-10912 was supposed to be resolved in 2.23. Could it be related to that anyway?

[timja]

daniel_c_686:

Sorry for editing this comment; I needed to fix some typos especially in my Jenkins version number.

I have this issue too on Jenkins 2.121.1, which I installed in the last week.
The symptoms are the same: increased memory usage at regular 5s intervals.
This only appears to occur on pages with the "Build Queue" and "Build Executor Status" widgets.

I have a lot of executors in my Jenkins instance: between 200 and 210. There's quite a bit of build activity because we have a large number of developers, so I see very quick memory growth.

I did a memory profile in Chrome of type "Allocation instrumentation on timeline" to see what was leaking.

I saw a lot of "(closure)", "Detached HTMLTableCellElement" and "HTMLAnchorElement".

Looking at the HTMLTableCellElements first, I saw some formatted job names. I worked backwards a bit to find that the were all inside "Detached HTMLTableRowElements".
Here's a sample of the outerHTML for one of those (item names changed, and whitespace/indentation added for clarity):

"pane" align="right" style="vertical-align: top">3
"pane">
"white-space: normal">
  "/job/my-folder/job/my-jobname/">My Folder » My Job Name
  "Started 20 hr ago
 Estimated remaining time: 10 hr" style="cursor:pointer" href="/job/my-folder/job/my-jobname/44/console" class="progress-bar " title="Started 20 hr ago
 Estimated remaining time: 10 hr">
    

    
"width:66%;" class="progress-bar-done">
    
"width:34%" class="progress-bar-left">
  

"pane">
  "/job/my-folder/job/my-jobname/44/" class="model-link inside">#44

"pane" align="center" valign="middle">
  "if(confirm("Are you sure you want to abort My Folder » My Job Name #44?"))new Ajax.Request("/computer/node%20name%20anonymized/executors/2/stop"); return false;" href="/computer/node%20name%20anonymized/executors/2/stop" class="stop-button-link">"/static/175f9ede/images/16x16/stop.png" alt="terminate this build" style="width: 16px; height: 16px; " class="icon-stop icon-sm">

Using the "collect garbage" option in Chrome did not remove these objects.
I looked at the (closure) objects to see what they were using the "Retainers" panel in the Dev Tools.

These closures are referenced from "Detached EventListener" objects, referenced by "Detached InternalNode" objects, referenced by "Detached HTMLAnchorElement" objects.

I did one more thing: "Break on subtree modification" for div#side-panel in the Chrome dev tools.

That brought me to hudson-behavior.js, "p.replaceChild(node, hist)", which offers some insight into why there are a tree of detached elements: some element inside the tree may have an EventListener that is still referenced somewhere.

I tried the Firefox Dev tools next (as well as to confirm that the memory leak also occurs in Firefox.

Tracing call stacks on memory allocations showed that there's an issue with the call stack starting at the very next function call, "Behaviour.applySubtree(node)":

The _createResponder function in prototype.js seems to be the culprit.
It seems to always add the passed element to its CACHE array. It never cleans up the CACHE as far as I can tell, unless you unload the page in IE.

Because the CACHE contains a persistent reference to the HTMLAnchorElements which get event handlers, all of the anchor elements created in the Build Executor Window are persisted. That adds up fast.

I hope that makes this bug fixable - I had a window where these were visible and it used up 3GB of memory after a few hours.

[timja]

trejkaz:

We're seeing something like this also. Overnight, my machine will get to 6~8GB per tab with Jenkins open in it.

I tracked it down to a cache in Prototype as well, but mine was 'cache' in lowercase instead of 'CACHE' in uppercase.

[timja]

trejkaz:

Here's a story by someone else with the same problem (though they discovered it in TeamCity, instead of Jenkins) who ended up forking Prototype to fix it.

http://kirblog.idetalk.com/2011/06/prototype-17-memory-leak.html

They have a patched version of prototype.js with a fix in it. Could this just be incorporated into Jenkins?

[timja]

trejkaz:

Is this the only copy of prototype.js to fix?

https://github.com/jenkinsci/jenkins/blob/master/war/src/main/webapp/scripts/prototype.js

[timja]

zioschild:

Maybe as fast workaround you can ignore the problem in IE and fix it for chrome and firefox by canging

    if (Object.isUndefined(registry)) {
      CACHE.push(element);
      registry = Element.retrieve(element, 'prototype_event_registry', $H());
    }

to

    if (Object.isUndefined(registry)) {
      if (!Prototype.Browser.IE) {
CACHE.push(element);
      }
      registry = Element.retrieve(element, 'prototype_event_registry', $H());
    }

This will prevent the elements getting stored in the array for 'not IE' browser...

BTW: there is an additional leak in Element.Storage in case of the element is not clearly deleted with the purge command. A workaround may be:

  getStorage: function(element) {
    if (!(element = $(element))) return;

    //var uid;
    if (element === window) {
      uid = 0;
    } else {
      if (typeof element._prototypeUID === "undefined"){
element._prototypeUID = Element.Storage.UID++;
      }
      uid = element._prototypeUID;
    }
    // -------------------
 //PATCHED: do not (!!) use Storage - see below
    //if (!Element.Storage[uid])
    //  Element.Storage[uid] = $H();
    //return Element.Storage[uid];
 
    // -------------------
 // PATCHED Storage is not cleaned correctly when not calling "purge" and leads to memleaks
 // for reusage we add the storage directly to the element
 if (!element._prototypeStorage)   
         element._prototypeStorage = $H();
    return element._prototypeStorage;
  },