Cookies not loading in v1.7, performance issue

[kilx]

Hi,

After updating to latest version the list does not display any cookies. It stalls at 10% doing nothing I guess. Regardless whether there's anything in the filter or not. Have a look at the screenshot.

Also, the menu in Tools is screwed up a bit, also in the screenshot.

http://imgur.com/AEBM1Hq

[MrWolf-to]

You must click "refresh". By the way, if I leave its window opened while browsing, it freezes Firefox for a while!

[kilx]

"Refresh" has no effect. I've waited for more than a minute.

[MrWolf-to]

Mmmm... I don't know, then. When I opened the window, it was blank, clicked "Refresh" and took a while to load cookies.

[vanowm]

What if you open CM+ via dropdown menu on it's button while a website is opened (aka with website filter)? Also, do you know how many cookies you have?

And finally, try change extensions.cookiesmanagerplus.debug in about:config to 7 and look in error console (CTRL+SHIFT+J) if any errors shown when opened CM+ (warning, it will flood the console with messages)

On my computer it takes about 2 seconds for almost 5000 cookies to finish loading.

[kilx]

Doesn't matter how or where I open CM+. Clicking refresh does nothing.

Immeditely when opening via drop down:

CM+ (sysOverlay.js:75:1): (string):github.com
CM+ (libOut.js:1:1): (debug):[libOut.js loaded] N/A execTime: 63104
CM+ (commonFunc.js:1:1): (debug):[commonFunc.js loaded] N/A execTime: 63105
CM+ (commonFunc.js:402:1): (debug):[startfalse] protect_init() execTime: 63107
CM+ (commonFunc.js:513:1): (debug):[startfalse] protect_CookieKeeper_init() execTime: 63108
CM+ (commonFunc.js:904:1): (debug):[commonFunc.js loaded] N/A execTime: 63111
CM+ (cookiesmanagerplus.js:293:1): (debug):[begin] start() execTime: 0
CM+ (html5.js:336:1): (debug):[html5.js loaded] N/A execTime: 4
CM+ (cookiesmanagerplus.js:764:1): (debug):[begin] onPrefChange_do() execTime: 18
CM+ (cookiesmanagerplus.js:2158:1): (debug):setFilter() execTime: 18
CM+ (cookiesmanagerplus.js:2211:1): (debug):setChecked() execTime: 18
CM+ (cookiesmanagerplus.js:2211:1): (debug):setChecked() execTime: 19
CM+ (cookiesmanagerplus.js:2211:1): (debug):setChecked() execTime: 20
CM+ (cookiesmanagerplus.js:2211:1): (debug):setChecked() execTime: 21
CM+ (cookiesmanagerplus.js:2382:1): (debug):[begin] infoRowsShow() execTime: 22
CM+ (cookiesmanagerplus.js:2440:1): (debug):[end] infoRowsShow() execTime: 3 [25]
CM+ (cookiesmanagerplus.js:2463:1): (debug):infoRowsSort() execTime: 25
CM+ (cookiesmanagerplus.js:817:1): (debug):[end] onPrefChange_do() execTime: 19 [37]
CM+ (cookiesmanagerplus.js:827:1): (debug):[begin] loadCookies() execTime: 37
CM+ (cookiesmanagerplus.js:849:1): (debug):[loading cookies] loadCookies() execTime: 38
CM+ (cookiesmanagerplus.js:897:1): (debug):[loading html5] loadCookies() execTime: 261
CM+ (cookiesmanagerplus.js:389:1): (debug):[end] start() execTime: 267 [267]
CM+ (html5.js:135:1): (debug):html5_load() execTime: 457

A while later:

CM+ (cookiesmanagerplus.js:1330:1): (debug):observe() execTime: 47875
CM+ (cookiesmanagerplus.js:1364:1): (debug):_handleCookieAdded() execTime: 47877

When clicking "refresh" several times:

CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 20546
CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 22226
CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 24296
CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 31170
CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 44676
CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 50692
CM+ (cookiesmanagerplus.js:1729:1): (debug):doLookup() execTime: 51620

Stalls at "10%", screenshot: http://imgur.com/c8AUsKB

The cookies files (cmd.exe dir command) in profile directory while FF is running:

2016-05-04  15:26         2,097,152 cookies.sqlite
2016-05-04  15:17            32,768 cookies.sqlite-shm
2016-05-04  15:17                 0 cookies.sqlite-wal
2016-05-04  15:24                 2 cookiesManagerPlus.json

The count I guess:

sqlite> select count(*) from moz_cookies;
2962

Anyhow the previous version was able to display them a couple of days ago. And a week ago or so before the upgrade of Firefox I was able to search and backup selected cookies without any issues.

List of enabled extensions:

Name: Adblock Plus
Version: 2.7.3
Enabled: true
ID: {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}

Name: Cookies Manager+
Version: 1.7
Enabled: true
ID: {bb6bc1bb-f824-4702-90cd-35e2fb24f25d}

Name: Element Hiding Helper for Adblock Plus
Version: 1.3.8
Enabled: true
ID: elemhidehelper@adblockplus.org

Name: Estonian ID Card authentication module
Version: 3.11.0.6179
Enabled: true
ID: {aa84ce40-4253-a00a-8cd6-0800200f9a67}

Name: Firefox Hello
Version: 1.2.6
Enabled: true
ID: loop@mozilla.org

Name: FoxyProxy Standard
Version: 4.5.6
Enabled: true
ID: foxyproxy@eric.h.jung

Name: Greasemonkey
Version: 3.8
Enabled: true
ID: {e4a8a97b-f2ed-450b-b12d-ee082ba24781}

Name: Multi-process staged rollout
Version: 1.0
Enabled: true
ID: e10srollout@mozilla.org

Name: Password Exporter
Version: 1.3.2
Enabled: true
ID: {B17C1C5A-04B1-11DB-9804-B622A1EF5492}

Name: Pocket
Version: 1.0
Enabled: true
ID: firefox@getpocket.com

Name: Tab Groups
Version: 1.0.2
Enabled: true
ID: tabgroups@quicksaver

Name: Tab Mix Plus
Version: 0.4.2.2
Enabled: true
ID: {dc572301-7619-498c-a57d-39143191b318}

Name: Tamper Data
Version: 11.0.1.1-signed.1-signed
Enabled: true
ID: {9c51bd27-6ed8-4000-a2bf-36cb95c0c947}

Name: Tree Style Tab
Version: 0.17.2016031101
Enabled: true
ID: treestyletab@piro.sakura.ne.jp

System info:

Windows 10 x64

Name: Firefox
Version: 46.0
Build ID: 20160421124000
Update Channel: release
User Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:46.0) Gecko/20100101 Firefox/46.0
Multiprocess Windows: 0/1 (Disabled)
Safe Mode: false

[vanowm]

It seems it stops while loading local storage data (html5 cookies). What if you try close browser completely, backup your profile folder and rename webappsstore.sqlite file in your profile folder to something else. See if it changes anything.

[kilx]

Deleting the webappsstore.sqlite file helped and it works OK now.

I tried VACUUM'ing all the .sqlite files in the folder including webappsstore.sqlite but that didn't change anything. I can live with the loss of this data. The file had grown to 41MB.

Related info here: https://www.reddit.com/r/firefox/comments/40gbos/psa_large_webappsstoresqlite_file_in_your_profile/.

Any point in investigating what entry caused CM+ to fail?

sqlite> select count(*) from webappsstore2;
3471

Also, I now see the CM+ menu in "Tools-> Cookies Manager+" correctly and the "Site: github.com" is displayed instead of a blank.

[Rikk]

Here it was very slow too, when searching cookies from local storage. I disabled the new option and it run just like old version. (Btw, why it took so long for an update? I was taking CM+ as an abandoned plugin, but still very useful sometimes ;)

My webappsstore.sqlite is around 90MB but CM+ didn't fail or anything, it was just taking too long to load that kind of cookies.

[Gitoffthelawn]

@kilx @Rikk Both of you have webappsstore.sqlite files that are exceptionally large. If you have sqlite skills (or even are good with a hex editor), can you tell us which sites are responsible for adding so much data to that file?

[Rikk]

@Gitoffthelawn The size of these files is controlled by Firefox. As being databases, their size is increased with a margin (empty space that is fulfilled as needed). They grow as you visit more and more sites; there also must be many metadata Firefox stores there. If you clean cookies and cache their size should decrease, at least should theoretically; but I think Firefox fails, or at least used to fail to clean some kinds of data however. The two fat databases here are webappsstore.sqlite and places.sqlite. But I don't think their size decreases performance. At least my x64 version looks ok.

[kilx]

Here's something:

sqlite> select length(value), scope from webappsstore2 where length(value) > 100000 order by 1 desc ;
2108342|moc.mvhh.scod.:http:80
2039244|ten.php.:http:80
1396653|gro.ymedacanahk.www.:https:443
1356039|gro.atadikiw.www.:https:443
916703|moc.elgoog.evird.:https:443
877471|moc.etals.www.:http:80
818612|gro.aidemikiw.snommoc.:https:443
769640|moc.krowemarfcinoi.:http:80
639244|gro.aidemikiw.5102ainamikiw.:https:443
560289|moc.sulpstut.ngisedbew.:http:80
549308|gro.aidepikiw.se.:https:443
530003|gro.aidepikiw.ed.:https:443
484408|gro.ikiwaidem.www.:https:443
406983|gro.aidepikiw.m.ne.:https:443
352517|ee.vmo.www.:http:80
351557|moc.rekilym.www.:http:80
345225|gro.aidepikiw.ur.:https:443
338313|moc.nideknil.www.:https:443
337968|gro.kcahefil.www.:http:80
314128|gro.skoobikiw.elpmis.:https:443
290104|gro.aidepikiw.elpmis.:https:443
282364|gro.aidemikiw.atem.:https:443
275884|gro.etouqikiw.ne.:https:443
264479|moc.gnimag2kcab.www.:http:80
262086|gro.aidepikiw.rt.:https:443
229569|moc.slicnepnez.:http:80
207647|moc.enixunil.www.:http:80
205396|moc.leccatib.www.:https:443
205396|moc.leccatib.www.:http:80
205396|moc.leccatib.:http:80
200924|moc.eslaf-neht-eurt-ton-fi.www.:http:80
192651|oi.elcitrap.dliub.:https:443
187982|moc.swenrekcaheht.:https:443
182792|hc.vtze.:https:443
182404|moc.keegyad2.www.:http:80
181009|moc.etsaphcet.www.:http:80
179812|gro.yranoitkiw.ed.:https:443
178358|moc.sepacstsop.:http:80
176051|ti.nioctib.ne.:https:443
174882|moc.eslaf-neht-eurt-ton-fi.www.:http:80
174860|moc.dtlxrew.:http:80
174332|moc.etsaphcet.www.:http:80
174240|moc.swenrekcaheht.:https:443
170974|moc.enixunil.www.:http:80
170695|moc.sakullam.www.:http:80
170680|moc.sakullam.www.:http:80
169667|moc.slicnepnez.:http:80
169242|moc.slicnepnez.:http:80
169073|gro.egnellahctxeneht.:http:80
165928|ee.piik.:http:80
163728|moc.swenrekcaheht.:http:80
163407|moc.gnippamegakcap.www.:https:443
161248|gro.egnellahctxeneht.:http:80
155318|moc.swenrekcaheht.:https:443
153158|moc.swenrekcaheht.:http:80
152937|moc.rekilym.www.:http:80
148836|moc.swenrekcaheht.:http:80
148504|moc.slairotut7.www.:http:80
143856|gro.egnellahctxeneht.:http:80
141367|ee.vmo.www.:http:80
138308|moc.xelf-asnah.tac.:http:80
137035|moc.sepacstsop.:http:80
136912|ee.vmo.www.:http:80
136426|moc.yfitops.ytinummoc.:https:443
133558|ee.vmo.www.:http:80
126715|gro.egnellahctxeneht.:http:80
124513|ua.moc.purettub.www.:http:80
123379|moc.elgoog.evird.:https:443
123362|moc.tsopnotgniffuh.www.:http:80
122793|moc.dirac.www.:http:80
119841|moc.sepacstsop.:http:80
119397|moc.swenrekcaheht.:https:443
118984|moc.gnippamegakcap.www.:https:443
118917|moc.slicnepnez.:http:80
118672|gro.egnellahctxeneht.:http:80
118445|moc.enixunil.www.:http:80
118046|moc.swenrekcaheht.:http:80
116768|gro.egnellahctxeneht.:http:80
116725|moc.swenrekcaheht.:http:80
116621|moc.arevteg.:http:80
115264|moc.dirac.www.:http:80
107922|moc.gnippamegakcap.www.:https:443
107717|moc.tciddasenohpdaeh.:http:80
106574|moc.gnippamegakcap.www.:https:443
106040|moc.gnimag2kcab.www.:http:80
105355|ee.vmo.www.:http:80
104359|gro.tcejorparodef.ksa.:https:443
103010|moc.slairotut7.www.:http:80
102417|moc.eslaf-neht-eurt-ton-fi.www.:http:80
102417|moc.dtlxrew.:http:80
100680|moc.slicnepnez.:http:80

And after a bit of work that's:

ask.fedoraproject.org
bitaccel.com
build.particle.io
cat.hansa-flex.com
commons.wikimedia.org
community.spotify.com
de.wikipedia.org
de.wiktionary.org
docs.hhvm.com
drive.google.com
en.bitcoin.it
en.m.wikipedia.org
en.wikiquote.org
es.wikipedia.org
eztv.ch
getvera.com
headphonesaddict.com
ionicframework.com
kiip.ee
meta.wikimedia.org
php.net
postscapes.com
ru.wikipedia.org
simple.wikibooks.org
simple.wikipedia.org
thehackernews.com
thenextchallenge.org
tr.wikipedia.org
webdesign.tutsplus.com
werxltd.com
wikimania2015.wikimedia.org
www.2daygeek.com
www.7tutorials.com
www.back2gaming.com
www.bitaccel.com
www.butterup.com.au
www.carid.com
www.huffingtonpost.com
www.if-not-true-then-false.com
www.khanacademy.org
www.lifehack.org
www.linkedin.com
www.linuxine.com
www.mallukas.com
www.mediawiki.org
www.myliker.com
www.omv.ee
www.packagemapping.com
www.slate.com
www.techpaste.com
www.wikidata.org
zenpencils.com

Many of the sites I don't recognize, then again the profile dates back to 2013 too. Still, FF should purge this periodically. It's still mostly temporary data.

Also, while Rikk said CM+ was finally able to load his cookie list that's not a case for me. I tried waiting for several minutes.

If anybody cares to write a query to sum the length(value)-s for each site I can run that too.

[Gatheringofravens]

Had the exact same issue as Kilx including all symptoms. Resolved by removing the webappsstore.sqlite file from firefox profile folder. Issue did not exist before current version, definitely something going on with this DB however. (And for reference my version of this file was 12.4MB, so not that large really)

(Unrelated but did you happen to reverse the urls from their left to right to proper right to left in the SQLite DB? If so, would you still have the command there for it? Viewing this database like this is annoying and I'm not very versed with sql)

Addons:
Addon                  Enabled?
Cookies Manager+    1.7.1   true    {bb6bc1bb-f824-4702-90cd-35e2fb24f25d}
Disconnect  3.15.3.1-signed.1-signed    true    2.0@disconnect.me
DownThemAll!    2.0.19  true    {DDC359D1-844A-42a7-9AA1-88A850A938A8}
Firebug 2.0.16  true    firebug@software.joehewitt.com
Firefox Hello Beta  1.1.14  true    loop@mozilla.org
Flagfox 5.1.10  true    {1018e4d6-728f-4b20-ad56-37578a4de76b}
FlashGot    1.5.6.13    true    {19503e42-ca3c-4c27-b1e2-9cdb2170ee34}
HTTPS-Everywhere    5.1.6   true    https-everywhere-eff@eff.org
Live HTTP headers   0.17.1-signed.1-signed  true    {8f8fe09b-0bd3-4470-bc1b-8cad42b8203a}
NoScript    2.9.0.11    true    {73a6fe31-595d-460b-a920-fcc0f8843232}
Policeman   0.18.1.1-signed.1-signed    true    policeman@futpib.addons.mozilla.org
Screengrab (fix version)    0.99.02c    true    {02450914-cdd9-410f-b1da-db004e18c671}
SSleuth 0.4.5   true    ssleuth@github
uBlock Origin   1.7.0   true    uBlock0@raymondhill.net
Web Developer   1.2.5.1-signed.1-signed true    {c45c406e-ab73-11d8-be73-000a95be3b12}
Tamper Data 11.0.1.1-signed.1-signed    false   {9c51bd27-6ed8-4000-a2bf-36cb95c0c947}

[Gatheringofravens]

Alright, I ran Kilx's sql code on the db I removed, here are the results (after reversing the text strings in excel):

select length(value), scope from webappsstore2 where length(value) > 100000 order by 1 desc ;
length(value)   scope   http
1091203 .en.wikipedia.org   :http:80
1071621 .de.wikipedia.org   :http:80
607410  .commons.wikimedia.org  :http:80
560211  .commons.wikimedia.org  :https:443
329689  .www.lifehack.org   :http:80
315326  .www.grainger.com   :http:80
243512  .www.grainger.com   :http:80
237112  .www.dillards.com   :http:80
236854  .ffrf.org   :https:443
236853  .ffrf.org   :http:80
229926  .www.digikey.com    :http:80
229544  .si.wikipedia.org   :http:80
220215  .thedailycrawfish.com   :http:80
202240  .www.lenovo.com :http:80
199530  .www.grainger.com   :http:80
181151  .ffrf.org   :https:443
177196  .www.addictinginfo.org  :http:80
175091  .www.addictinginfo.org  :http:80
175051  .www.addictinginfo.org  :http:80
173474  .thehackernews.com  :http:80
155317  .thehackernews.com  :http:80
154836  .ffrf.org   :https:443
146457  .www.theguardian.com    :http:80
131449  .www.grainger.com   :http:80
123362  .www.huffingtonpost.com :http:80
122212  .drive.google.com   :https:443
121940  .www.theguardian.com    :http:80
119397  .thehackernews.com  :http:80
117593  .www.theguardian.com    :http:80
104997  .berniesanders.com  :https:443

Interestingly some of these websites I've only visited a couple of times, at least one I think I've probably only visited once.

[Gatheringofravens]

Alright, I have the data from the first one (en.wikipedia.org, the biggest by far of several entries for it), it's too huge to post in a comment here (It's over 1mb of text) so I'll link to pastebin zerobin (since pastebin has a 512kb limit) Scope: .en.wikipedia.org :http:80 Key: MediaWikiModuleStore:enwiki Value: https://zerobin.net/?c58746f3e7ea85a9#3vzzqCKU5mhXmEotbQMOMNFyfid6+OtdcO+PkdWxins=

(I don't think there is much private data here but just the same copy it and paste into a notepad document as I have it set to delete in one week)

If I had to guess I'd say maybe these huge values or something about them is causing it to choke?

[MrWolf-to]

My webappsstore.sqlite file was 56 MB!!! I don't know what was in it. I deleted it, Firefox seems a bit faster (I have a SSD) and Cookie Manager+ doesn't hang Firefox anymore.

[vanowm]

Looks to me, like websites are abusing this local storage feature. In Gatheringofravens wikipedia entry it's a storage for javascript it used. In theory it makes sense, it's faster to load stuff from local computer then from a webserver, but apparently it creates more issues then solves any..

@kilx try e41d77c I might have found the bug why it stops loading cookies. (the only change in this version is in html5.js on line 165)

[OFFTOPIC] As of domain name in revers - that's is how Firefox decided to store data. I don't know what were they smoking that they thought it was is a good idea, but perhaps the same thing when they decided to discontinue XUL support in very near feature (the backbone of FF) which will kill 99% of all extensions, including CM+, making FF nothing more but a Chrome clone.

[kilx]

It was able to load the cookies now, though bloody slow. Each search takes some 12-17 seconds and FF main window freezes sometimes for a few seconds when the CM+ window is left open as if it's doing another search a bit later.

That's on a Thinkpad x230 with a SSD, i5-3230M@2.6GHz and 16GB RAM on Win10 x64.

I could not find a handy reverse() function in sqlite either.

[hkmaly]

Me too, I mean my webappsstore.sqlite is 49MB and refreshing the list of cookies is painfully slow after upgrade. Biggest cookies are the PHP ones (www.php.net, php.net)

Will this be fixed or is the API slow by design?

Note: The domain name in reverse is because you can then easily search for subdomains. I mean, on some.example.com, cookies for examples.com and some.examples.com are relevant, and you can get both with query for 'moc.elpmaxe%' which is indexed and therefore usably fast. So, nothing like the "kill XUL" decision. BTW, I will be switching to palemoon.

[vanowm]

Is it still slow when extensions.cookiesmanagerplus.html5 setting in about:config is set to false?

P.S. Thanks for the reversed domain explanation. That makes sense.

[Rikk]

Also, the menu in Tools is screwed up a bit, also in the screenshot. http://imgur.com/AEBM1Hq

I'm also curious what should appear inside that new sub-menu, v1.5x didn't have it.

[vanowm]

If the menu is not complete, that means the cookies failed loading.

On my computer with almost 4000 cookies when html5 is disabled CM+ opens and finishes loading almost instantly. With html5 enabled it takes 2 seconds longer, but my webappstore.sqlite is only 11mb So, I think for now I'm going to disable html5 by default. I'm baffled why it freezes up whole browser though, I thought async should help with that...

[jd--]

In my case problem was that in webappsstore.sqlite one "key" field is blank. After I set that key field to some arbitary value everything seems fine.

Just for testing I blanked some other key field and problem reapears. Then assigned some value to that key field and everything is fine again.

[MrWolf-to]

What program do you use to open webappsstore.sqlite? I have an addon for Firefox: "SQLite Manager", but, when I open it, I don't know what to search... Then, I found "SQLiteStudio": when I open the file, I find a bunch of data, but I can't understand them.

[jd--]

In SQLiteStudio click on Data tab and you can see key:value pairs for various sites. Note that sitenames are in right-to-left notation ;) You can double click to edit fields but you need to click on green checkmark to save changes to DB.

If you have some blank "key" fields you can change them to avoid this issue. Deleting that row also works.

Edit: As for "bunch of data": it's just that bunch of data bits that site is storing on your computer for example your user id (key: id; value: 1785421) or for this site: loged-in = true ;)

[Rikk]

If the menu is not complete, that means the cookies failed loading.

Here it doesn't show, with or without html5 cookies enabled, and it even shows a number of cookies higher than SQLite Manager (thanks for the tip, it's a very handy addon!) for cookies.sqlite.

[vanowm]

Strange that blank key creating issues for you, because in html5.js it skips these entries entirely. I've tested with a blank key and CM+ simply didn't show that entry, no performance change at all.

As of sub menu, first of all, I thought we were talking about tools menu inside CM+ window... In the browser's tools menu it should show the site address loaded in current tab or nothing if opened non http type address.

P.S.Lets try keep this issue on topic, if you have unrelated issue/question, start a new topic. Thanks.

[Rikk]

Menu issue is on the first post, so I think it is on-topic. Console display:

TypeError: m is null    sysOverlay.js:78:4
NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIURI.host]      sysOverlay.js:75:0

I would say it's failing on the host = gBrowser.currentURI.host; Maybe it is updating menu prior to loading the page?

[vanowm]

Fixed in v1.7.2b2

[jd--]

@vanowm I was using latest stable version (v1.7.1) and this blank key field bug is fixed in v1.7.2b1

You can enable local storage support by default ;)

[kilx]

Is it still slow when extensions.cookiesmanagerplus.html5 setting in about:config is set to false?

No, that speeds it up a lot. There's still a delay of about a second, but that's OK.

Sorry for the late reply.

I guess it's OK to close this issue now?

[vanowm]

Submitted v1.8, with removed support for HTML5 cookies, it significantly improved performance. v1.7.x moved to beta branch

Another thing I've noticed that kills the performance is CookieKeeper, when a lot of cookies are protected.