memory leak with extended usage

[silverwings15]

hi, i noticed that after installing requestpolicy (beta), my browser (Pale Moon 25.2 x64) would slow down dramatically after about 30-40 minutes of usage. about:addons-memory reports that it uses over 30 MB of memory, versus ~3MB after startup. my laptop has 6GB of RAM so computer specs isn't a problem, but the lag is severe enough to render the browser near unusable without a restart.

would be swell if you guys can look into this, and thanks for the stellar job on what is otherwise an amazing addon

[myrdd]

Hi @mimecry where did you read the 30MB? In my case I've got an entry called requestpolicy@requestpolicy.com/js-non-window/zones/zone(0x7fb08ac91000). When I click on it, it shows me details. Could you please tell me those details, at the time the browser is slow? In my case it looks like this:

│   ├───5.98 MB (01.21%) -- requestpolicy@requestpolicy.com/js-non-window/zones/zone(0x7fb08ac91000)
│   │   ├──3.34 MB (00.67%) -- compartment([System Principal], resource://requestpolicy/Ruleset.jsm)
│   │   │  ├──3.02 MB (00.61%) -- objects
│   │   │  │  ├──2.27 MB (00.46%) -- gc-heap
│   │   │  │  │  ├──1.54 MB (00.31%) ── ordinary
│   │   │  │  │  ├──0.71 MB (00.14%) ── dense-array
│   │   │  │  │  └──0.02 MB (00.00%) ── function
│   │   │  │  ├──0.75 MB (00.15%) ++ malloc-heap
│   │   │  │  └──0.00 MB (00.00%) ── non-heap/code/asm.js
│   │   │  ├──0.30 MB (00.06%) ++ shapes
│   │   │  ├──0.01 MB (00.00%) ── scripts/gc-heap
│   │   │  ├──0.01 MB (00.00%) ── type-inference/type-scripts
│   │   │  └──0.01 MB (00.00%) ++ sundries
│   │   ├──0.79 MB (00.16%) ++ compartment([System Principal], resource://requestpolicy/RequestUtil.jsm)
│   │   ├──0.78 MB (00.16%) ++ compartment([System Principal], resource://requestpolicy/RequestResult.jsm)
│   │   ├──0.23 MB (00.05%) ++ compartment([System Principal], resource://requestpolicy/RequestProcessor.jsm)
(...)

So in my case Ruleset.jsm needs most of the memory.

[silverwings15]

thanks for the reply! i installed the addon (along with my others) on a brand new profile to make sure that the lag is not caused because of other reasons. much better now, the highest the addon's gotten so far is ~20 MB, and that's with very extended usage. though i noticed it is quicker to rise when i'm using a lot of Flash (watching multiple livestreams simultaneously). according to about:memory, the bulk of memory usage comes from RequestResult.jsm (7.43 MB out of 20.18) and RequestUtil.jsm (8.70 MB out of 20.18). i'll report back if the old behavior resurfaces

[myrdd]

Ok, thanks for the details! I suppose that when viewing videos you're getting many requests. Could you please check the menu how many requests you can see when the browser is slow? — see also https://github.com/RequestPolicyContinued/requestpolicy/issues/475

Besides that: how many rules do you have in your "normal" profile? Which of the subscriptions do you have enabled? Is the browser getting faster if you disable the subscriptions? — the official-deny_trackers.json subscription recently got a lot of new rules (before: ~100, afterwards: >2000)

[silverwings15]

sorry for the late reply. how can i check for these values?

[myrdd]

The number of requests for an origin or destination you can see in the menu — to see them you need to enable the option in the Advanced Preferences (chrome://requestpolicy/content/settings/advancedprefs.html)

The rules that are active for your profile you can see on chrome://requestpolicy/content/settings/yourpolicy.html

[silverwings15]

hi again. is this what you're asking for?

http://i.imgur.com/JCu4Ujw.png

http://i.imgur.com/9tvZfuh.png

^ my two flash-using tabs (screenshots taken as RP's memory usage totals 21.47 MB). i have about 30 manual rules. on the subscription policies page, i don't see any options aside from a checkbox for Privacy: Block destinations that can track you as you browse the web

[myrdd]

Yes, that's what I've been asking for. Interestingly the counter is at 327 and 206, respectively – not that much.

on the subscription policies page, i don't see any options aside from a checkbox for Privacy: Block destinations that can track you as you browse the web

is that checkbox activated? Could you please check whether there is any significant difference in performance when you activate/deactivate that checkbox? Thanks for your help!

[silverwings15]

the value on the joindota site varies - some pages go as low as 100 and others will be 600 (all flash streams). not sure why that is the case, as those pages are identical save for which stream is embedded

yes it is activated. i'll report back once i've had sufficient time browsing with that option unchecked

[silverwings15]

i don't notice any difference in performance after turning off the option. however i think i've located the issue - the number of requests for a flash-using tab gradually increases overtime. at the moment it initially finishes loading, it varies between 100-200. this is what it looks like after roughly 30 minutes

http://i.imgur.com/H0LJNVl.png

at that point i had to reset, as the browser lag became too much

[myrdd]

Here we go, more than 2000 requests. This underlines that the mechanism of saving rules causes those performance issues. Thank you for your help tracking down the problem, that was great help!

As I've said before, issue #475 is related to this one.

[silverwings15]

yes, it was reading it that i got the idea to check the counter after leaving the tab open for an extended duration, instead of at the first sign of lag

glad i could help! i even had to disable the addon to ensure that i wasn't unintentionally misleading you, but requestpolicy was indeed the source of the lag, because my browser resumed functioning regularly with it disabled

[nodiscc]

I can confirm having a lot of cross-site requests causes an overall slowdown. Slowdown when opening the menu is obvious, I can also feel less responsiveness in page navigation. This happens quite a lot with my RSS aggregator (TinyTinyRSS) on which there is a lot of embedded content from feeds I'm subscribed to:

[myrdd]

Another report by @sander85, copied from #672:

It's OK at first but with few days running it's memory usage grows and grows until Firefox becomes slow to use and has to be restarted. Doesn't happen if this add-on is disabled.

Output from about:memory

├─────67.59 MB (05.67%) -- add-ons
│     ├──58.08 MB (04.87%) -- requestpolicy@requestpolicy.com
│     │  ├──57.13 MB (04.79%) -- js-non-window/zones/zone(0x7f8e2d42e800)
│     │  │  ├──26.14 MB (02.19%) -- compartment([System Principal], chrome://requestpolicy/content/lib/request-result.jsm)
│     │  │  │  ├──23.63 MB (01.98%) -- classes
│     │  │  │  │  ├──17.12 MB (01.44%) -- class(Array)
│     │  │  │  │  │  ├──17.12 MB (01.44%) -- objects
│     │  │  │  │  │  │  ├──17.12 MB (01.44%) ── gc-heap
│     │  │  │  │  │  │  └───0.00 MB (00.00%) ++ malloc-heap
│     │  │  │  │  │  └───0.00 MB (00.00%) ++ shapes
│     │  │  │  │  └───6.51 MB (00.55%) ++ (3 tiny)
│     │  │  │  └───2.51 MB (00.21%) ++ (2 tiny)
│     │  │  ├──20.47 MB (01.72%) ++ compartment([System Principal], chrome://requestpolicy/content/lib/request-set.jsm)
│     │  │  └──10.53 MB (00.88%) ++ (31 tiny)
│     │  └───0.95 MB (00.08%) ++ window-objects/top(none)/detached/window(about:requestpolicy?yourpolicy)

In this case, especially request-result.jsm has grown.

[sander85]

I have pretty BIG numbers in the allowed requests counters. One is easily reproducible @ www.postimees.ee - they have the news ticker at the top right corner. It's loading images with ajax. Current number is 4000+ from postimees.ee to pmo.ee The other big one is Google+ and their googleusercontent.com which has 4300+ allowed requests. Both sites are quite SLOW to use :) and opening RequestPolicy's menu is even slower..

Question: if i read news @ postimees.ee and the counter gets reset to 0, does it release the memory or still not? If not then there are probably tens of thousands of requests logged :/

[myrdd]

Yes, 4000+ requests is a lot, but maybe quite normal for a page using XHRs a lot.

Question: if i read news @ postimees.ee and the counter gets reset to 0, does it release the memory or still not? If not then there are probably tens of thousands of requests logged :/

It should release the memory for that specific origin, e.g. http://postimees.ee/abcd.html. This way you should be able to free some memory, but I guess that closing the tab doesn't help.

I think there are several issues:

• Too much data (information) is stored about the requests (in general).
• Closing a tab does not remove the associated data.
• Information about background requests is stored as well, but never used. Background Requests aren't associated to any Tab.

[sander85]

I'd propose to create some loglevels, where the current logging would be something like debug and for normal use it would log almost nothing. If user needs to debug something s/he would turn on debugging.

[myrdd]

@sander85 is your request related to this memory leakage? If not, please create a new issue, or write on the general discussion. I am interested in what information you expect to be logged. There are already some log levels, see here. I think SEVERE and WARNING would be suitable, maybe excluding INTERNAL. Edit: see also issue #5

[sander85]

Yes, it's very much related to this memory leakage. I usually keep my browser open from update to update. But this add-on makes it currently pretty much impossible :( I watched F1 live timing today and there were many requests in the background. The memory usage is now @ 80MB for this extension and GC is stalling my browsing :P Also the CPU usage is growing. There are probably not too many users who keep their browser running for 24/7 but as I have quite a lot of tabs open (and some ask for password every time when I open my browser) then it would be a pain to restart it every day. This extension is still a must, so not sure what to do :)

Thanks for investigating!

[myrdd]

Yes, it's very much related to this memory leakage.

How? Do you want to see in the logging what is causing the memory leak?

The memory usage is now @ 80MB for this extension and GC is stalling my browsing :P Also the CPU usage is growing.

There's no doubt, this is too much.

There are probably not too many users who keep their browser running for 24/7

I wouldn't underestimate long-running browser sessions. I am as well restarting only if necessary.

I could think of a hack:

• a page for experiments: about:requestpolicy?experimental
• it will show how many request objects exist
• it allows to remove all those objects

What do you think? I think it would solve the problem for the time there's no real fix for this problem. You'll simply use that page, and (if it works) all memory leak will disappear. Let me know what you think and I'll schedule it.

Edit / Note to myself: the "free memory" feature will reset not only the RequestSets, but also _clickedLinks etc. – see performance-related issues.

[sander85]

Yeah, I would definitely try it! Seems a lot easier to restart my whole browser :)

And I probably blamed logging in the memory leak, that's why I wanted to turn it off. If it's not so then never mind.

[myrdd]

And I probably blamed logging in the memory leak, that's why I wanted to turn it off. If it's not so then never mind.

I see! Logging shouldn't impact memory usage, and it's off by default. The Request Log (not to be confused with Logging) on the other hand does consume memory, but only if it's open.

I would definitely try it

Ok, please follow issue #673.

[myrdd]

Issue #673 has been implemented in the latest nightly. Just visit about:requestpolicy?experimental. @mimecry @sander85 I'd be glad to hear your experience