[RESOLVED] v57: uBo settings affected by clearing Offline Website Data on close.

grauenwolfe commented 6 years ago

Issue Resolved by deleting and reinstalling v57. No idea what happened...

I have always had Offline Website Data checked (value = true) for both privacy.cpd.offlineApps & privacy.clearOnShutdown.offlineApps. Since v57 this causes uBo to lose it's settings just as it was doing before adding an exception in v56.

Additionally, I get this in Preferences. Notice the Site Data info. It permanently reads "Calculating" and the options are unavailable.

fu ff

I do NOT want to start allowing a bunch of BS to get stored in order for extensions to work like they have all the years before.

Anyone else getting this? I imagine we all have relatively similar settings.

grauenwolfe commented 6 years ago

Made a new profile from scratch, not one thing was imported, and have gone through implementing the changes I have always had, item by item. Still a problem on new profile.

It seems that data from extensions is treated no differently than offline website data. Either Prefences>Cached Web Content -or- Offline Website Data on close is clearing extension data, uBo at least. Adding exceptions makes no difference anymore. Maybe something unique to only uBo? Universal? Shit.

if you clear OWD and NOT cookies what happens to profile/storage/default/moz-extension+++UUID dirs and profile/browser-extension-data/Ext-ID dir contents (this is not the UUID, eg see uBlock0@raymondhill.net folder)...

Not clearing cookies at all, Cookies + Site Preferences are left unchecked, not cleared. Interesting to note that profile/browser-extension-data/uBlock0@raymondhill.net/storage.js has a creation time of a few minutes ago, 13:05. While New Tab Override's storage.js file shows creation at 8:09, when I created a new profile.

pref("browser.storageManager.enabled", true); pref("dom.storageManager.enabled", true);

These are both enabled so we can rule that out.

might be something to do with browser.cache.offline.enable

This is my hunch as well. About to try more things out. What a f'ing mess, again. I genuinely may just go back to v56. Three days of using v57 and nothing is better or faster, only slightly more broken and glitchy, to be expected though I guess.

grauenwolfe commented 6 years ago

I'm 99.9% sure this is caused by clearing Offline Website Data. What I'm not at all sure of is if it's an issue only with uBo or if it is universal.

Below configuration solves the issue. Bad solution.

privacy.clearOnShutdown.offlineApps = false privacy.cpd.offlineApps = false

claustromaniac commented 6 years ago

I have both privacy.clearOnShutdown.offlineApps and privacy.cpd.offlineApps set to true and I'm not experiencing this on FF 57 stable. :confused:

claustromaniac commented 6 years ago

@Thorin-Oakenpants Yup. It must be that.

grauenwolfe commented 6 years ago

My Q was

if you make the two storage api prefs in 2706 false does it still happen
also, I am still no closer to knowing if it removes extension local storage and/or IDB storage

Shit man, I'm sorry, you're talking about:

dom.storageManager.enabled + browser.storageManager.enabled

OK, they were set to true. Switched them to false so we'll see what happens.

PS - If you're still talking about different entries than this then just tell me specifically. :)

Have fun wherever you're headed to.

Update: Makes no difference. Still removing ext prefs and settings every launch.

earthlng commented 6 years ago

I've tested a fresh new profile with only my user.js and a new install of the latest uBO and clearing offlineApps on shutdown (or manually) doesn't remove my uBlock config in 57 stable. IDK what to tell you, must be something on your end. Am still trying to figure out the "Calculating" part.

earthlng commented 6 years ago

Ok, so ... the "Calculating" part is because of browser.cache.offline.enable=false. Changing it to true creates a new folder OfflineCache which can also be seen under about:cache -> appcache -> location: profile\OfflineCache. Keeping it on true and setting browser.cache.offline.capacity = 0 changes that "location" to "none, only stored in memory". However then the Site Data UI thingy also doesn't work, .capacity needs to be at least =1.

from nightly's sanitize.js:

offlineApps now clears:

AppCache - which seems to be what's stored under profile/OfflineCache -> I've tested the IDB testpage and about:cache still said "storage in use: 0 KiB" - IDK which API is used for this AppCache stuff
LocalStorage - (or DOM Storage as Pants likes to call it) which I think is stored in webappstore.sqlite -> API: window.localStorage (sometimes also used in Webextensions)
ServiceWorkers
QuotaManager stuff - which is IDB and presumabely other stuff stored in profile/storage/...

When I tested the IDB testpage the Site Date Preferences UI now reads "Your stored site data is currently using 48.0 KB of disk space" and the domain is listed under "Settings..."

"Clear All Data" deletes it, as expected, HOWEVER manually clearing "Offline Website Data" does not remove it!?! WTF?!

console shows:

Error sanitizing offlineApps Exception { name: "NS_ERROR_ILLEGAL_VALUE", message: "Component returned failure code: 0x…", result: 2147942487, filename: "chrome://browser/content/sanitize.js", lineNumber: 336, columnNumber: 0, data: null, stack: "clear/</</<@chrome://browser/conten…", location: XPCWrappedNative_NoHelper }  sanitize.js:152
Error: Error sanitizing  sanitize.js:184:13

Also didn't clear it on shutdown - good stuff

EDIT: privacy.firstparty.isolate=true is what's causing the problem.

earthlng commented 6 years ago

I suggest we do the following changes for 57:

user_pref("browser.cache.offline.capacity", 1);
user_pref("browser.cache.offline.enable", true);
user_pref("browser.storageManager.enabled", true);
user_pref("dom.storageManager.enabled", true);
user_pref("privacy.firstparty.isolate", false);

Then the new Site Data UI works and offlineApps (Offline Website Data) clearing on shutdown or manually works correctly. A bit sad to disable FPI but IDK what else to do. What do you guys think?

edit: sanitize.js is not origin-aware in the latest nightly, see this vs _removeQuotaUsage. That's why "Clear All Data" works but clearing it on shutdown and via "Clear Recent History" does not.

grauenwolfe commented 6 years ago

PREFACE: As a matter of respect and efficiency I try my best to convey only relevant info, in as few words as possible. None of it is extraneous so please be sure to read and consider every word. I only say this because (a) it's a long post and (b) I'm guilty of first jumping to bullet points, screen shots, etc.

Now, on with the show...

I spent all night trying to isolate this. I toggled more settings in one night than anyone ever should but in the end the only single thing that mattered was clearing OWD. Either via History>Clear Recent History or via clear on close, it didn't matter. I toggled these one off one on, back and forth, etc. and just as you may expect they had the same effect individually or paired. When both clear OWD settings are off, everything is fine. No progress there - only confirmed what I already knew.

Issue 1

Setting browser.preferences.offlineGroup.enabled = true, adds this to the Privacy & Security UI:

offlinegoup

It shows "using 0 bytes of disk space". So I changed some extension settings, visited a few sites, and refreshed the Privacy page to see if it changed. It didn't, still "0 bytes". Yet when I clicked the "Clear Now" button, uBo settings are wiped clean - even though it says there's nothing there to clear.

Issue 2

Within Privacy & Security UI, three seperate references for storage show the following:

Cached Web Content = 0 bytes
Site Data = 0 bytes
Offline Web Content and User Data = 0 bytes

However, at the very same time:

ubopermissions

You can clearly see Offline Storage is 30.9 MB. Add to this that clicking either the "Clear Now" (via Offline Website Content UI) or the "Clear Storage" (via Permissions Manager UI) buttons had the same effect on the same data even though they show inconsistant values.

Can I also just say that I have never been asked to allow a site storage permission. Not one time.

Quickie reminder points:

This issue started with v57 on a profile which I have used trouble free since v53 or 54.
Issue persists on new profile from scratch without importing a single item from the previous profile to eliminate chance of corruption.

grauenwolfe commented 6 years ago

@earthlng I read your posts, not dismissing them, I just had a post ready to go.

Aside from my minor problem, it seeming like there's still some funky issues with storage overall. I read through some bugzilla tickets regarding storage last night and it quickly become apparent that sometimes the devs themselves are unsure about data storage - what gets stored, where, or what gets cleared and by what means.

I'll get back on this later today and of course post any progress made.

earthlng commented 6 years ago

First of all LOL at the preface xD It wasn't that long of a post. Maybe for today's youth it is because they won't read a tweet if it's longer than 1 sentence but that's another problem entirely ;) 2ndly just forget about browser.preferences.offlineGroup.enabled! That's old and probably won't work correctly anymore or could even interfere with the new "Site Data" UI. Reset it to false and see if that changes anything. 3rdly I also didn't have to change any permissions for uBlock and I can clear cookies, OWD, and everything else for that matter and it still retains uBlock's config. Regarding the "0 bytes" throughout that's the same here. It seems that storage used for extensions is not included in "Site Data"'s counter which also makes sense IMO. I assume you're using the latest version of uBlock? Have you tried making a couple screenshots of the settings and copying the filters and rules and completely remove and reinstall the latest version and then re-configure it? @claustromaniac couldn't reproduce your issue and neither could I, maybe it's a Mac problem IDK.

grauenwolfe commented 6 years ago

@earthlng Before starting this post I implemented the recommended changes so after posting this I'll restart FF and see where it goes (which may end up making this post pointless then).

claustromaniac couldn't reproduce your issue and neither could I, maybe it's a Mac problem IDK.

I understand that it's more than likely an issue for only me or maybe very few, but it's not been a problem before as I've cleared OWD for as far back as I can remember. Certain settings are sort of second nature for me at this point. It started immediately after updating to v57. From some tickets I read last night and this one 1416219 referenced by Pants, I know it's at least real even if contained.

I assume you're using the latest version of uBlock?

Using v1.14.18 - latest as far as I know.

Have you tried [...] completely remove and reinstall the latest version and then re-configure it?

Yes. I tried a number of things before posting and this was the second thing I did, actually. It didn't help so the next step was creating a new profile. It also didn't help.

3rdly I also didn't have to change any permissions for uBlock and I can clear cookies, OWD, and everything else for that matter and it still retains uBlock's config.

(Assuming this excludes permissions asked for on install) you haven't granted any permissions or added any exceptions? Would be nice if that was only necessary for v56.

Couple of quick questions for anyone:

Is your uBo data in profile/storage/default -or- profile/storage/permanent?
Did you update or clean install v57?
Do you have anything in your profile folder named webappsstore2

claustromaniac commented 6 years ago

@grauenwolfe

profile/storage/default.
I updated to v57, but I had previously clean-installed v56 if I remember correctly. It might be relevant.
No. EDIT: Wait. Was that a typo? I do have webappsstore.sqlite, for whatever it's worth.

earthlng commented 6 years ago

(Assuming this excludes permissions asked for on install) you haven't granted any permissions or added any exceptions?

No, but I use network.cookie.cookieBehavior;1 because I don't need any permanent cookies (and I clear them on shutdown). So perhaps you still need cookie-exceptions if you block them with that pref IDK.

profile/storage/default
clean install (portableApps)
no profile folder with that name. webappsstore2 is the name of the database table in the file webappsstore.sqlite. That bugzilla is about problems with extensions that still use the window.localStorage API but I'm pretty sure uBlock doesn't use that anymore. The new storage API that WE's should use is browser.local.storage which creates the files in profile/browser-extension-data/extension-name/storage.js. uBlock also stores some of its data in IndexedDBs in the folder profile/storage/default/extension-UUID/...

grauenwolfe commented 6 years ago

Edit: Added some related tickets. Weird this isn't a widspread problem so far.

No. EDIT: Wait. Was that a typo? I do have webappsstore.sqlite, for whatever it's worth.

Not a typo, webappsstore2 is correct but @earthlng cleared this up. Just me grasping at straws.

No, but I use network.cookie.cookieBehavior;1

Same here.

no profile folder with that name. webappsstore2 is the name of the database table in the file webappsstore.sqlite.

Ohhhh, ok. Got it.

Unfixable for now

As a final test I tried this once more:

created new profile via command-line, set as default
launched Firefox with new profile
immediately to AMO and installed uBlock0
opened uBlock0 dashboard and updated default lists
cleared History (default checkmarks) - settings remain
cleared History (default checkmarks + OWD) - settings gone

So it's time to call it on this issue.

Maybe something got off when updating? Don't know, could be a million things. Having an untouched, pristine profile still do it - it's beyond me. Maybe I'll just reinstall from scratch at some point. I'll deal with it but ultimately just wanted to understand the issue more than anything.

2glops commented 6 years ago

@grauenwolfe

default/storage/default/
update from my old regular profile
webappsstore.sqlite, webappsstore.sqlite-shm, webappsstore.sqlite-wal

No issue wit uBO.

grauenwolfe commented 6 years ago

from @grauenwolfe 's ridiculously excessively long post xD ..

And I tried so hard ... :)

so cookies are allowed by default

Yes. That list is verbatim and literal. 1. Launched FF and 2. immediately went to AMO. No setting prefs or anthing else in-between the steps posted.

did ALL settings vanish - not just lists back at default

Wednesday night I deleted FF thoroughly and installed v57. Fresh install is still doing it, sort of. My filters, rules, etc., are fine but it still won't keep uB0's dashboard settings. This is minor although annoying. For example, if I collapse a filter group it will stay collapsed indefinitely until OWD is cleared. So I'm still kind of left wondering what's going on. I would think if anything that maybe clearing "Site Preferences" would be responsible, not OWD. Examples screen shots below.

Dashboard set like this:

screenshot

Reverts to this after OWD is cleared:

screenshot 1

So, that's where it stands as of now. Just cosmetic really and I can completely live with this. Still annoys the piss out of me though since I can't figure out why.

grauenwolfe commented 6 years ago

PREFACE - PART II: This is mostly a rant. No, it's entiely a rant, actually. So incredibly frustrated with Firefox and although I know better than to email or post when angry, I'm doing it anyway.

Thorin, I agree with you entirely and I'm done.

Storage management is broken / implementated to poorly to be of any use. In a number of those tickets, the devs themselves are often confused by this monster they've created, what hope have I then? I've seen enough tickets, enough contradictions, and enough of Mozilla's amateur hour hackshow. I don't think answering your follow-up questions will matter since I don't think there's anything for us to figure out, it's broken. Think of Jack Skellington doing all of his tests and experiments trying to figure out Christmas...

I'll leave you with this: To solidify my feeling that 'managing storage' in Firefox is a sham, I disabled every storage, cache, disk, capacity, you-name-it, option. Set them all to off, disable, 0 value, etc., and guess what, none of it mattered, not one damn bit. Defying belief, data was still stored and data was still there to be deleted. So, either there are dozens of hidden preferences or it really is just a facade and sham.

Even with me clearly saying "No Firefox, don't do this." Firefox just looked at me and said "What? You mean this?" as it continued to store and cache away.

A full week dedicated to this only to find that settings don't mean shit. Best guess: it's unfixable and probably because it has never really been properly implemented to begin with.

gorhill commented 6 years ago

This is a long thread, I don't know if this was clear so I will just state it here:

Reverts to this after OWD is cleared

These UI settings are saved into uBO's window.locaStorage. The filter lists' contents and their compiled versions are saved into uBO's window.indexedDB. The rest goes into WebExtensions browser.storage.local (what is saved when backing up uBO settings).

earthlng commented 6 years ago

profile\browser-extension-data is browser.storage.local and window.localStorage is in webappsstore.sqlite

grauenwolfe commented 6 years ago

@gorhill Hey, the man himself.

This is a long thread, I don't know if this was clear ...

It's pretty long so probably best to ask you this:

What about:config prefs need to be left alone or are "off-limits" for uBO to function perfectly as designed? Anything like dom.indexedDB, browser.cache.offline, dom.storage, any other storage, any particular caches or anything else along these lines that uBO must have acces to? Or does it get access to everything it needs through asking for permissions when installed?

Maybe this is just a temporary condition with v57, I don't know, but uBO is an incredibly important extension for me so if there's anything you suggest, it'll be done.

grauenwolfe commented 6 years ago

Yes, correct. Since deleting and fresh installing 57 from scratch, filter list display issue now only. It doesn't affect uBO's functionality, just a minor annoyance.

uBO is working great since reinstalling, conveyed in post 346779584, maybe not clearly enough.

Wednesday night I deleted FF thoroughly and installed v57. Fresh install is still doing it, sort of. My filters, rules, etc., are fine but it still won't keep uB0's dashboard settings. This is minor although annoying. For example...

Before that (installed update from 56) was a complete nightmare. Updating filter lists over and over, 3p scripts enforced/not enforced randomly, wierd stuff and probably something funky during the update process, not sure.

earthlng commented 6 years ago

@grauenwolfe can you please update your original post with the solution (did you only have to re-install FF or also start a new profile?), and a note that this happened on a Mac. You can link to this comment https://github.com/ghacksuserjs/ghacks-user.js/issues/273#issuecomment-345508296 re: the "calculating part". And then please close this issue. Thanks

grauenwolfe commented 6 years ago

@earthlng Sure, good idea.

grauenwolfe commented 6 years ago

@Thorin-Oakenpants Only just now, way late (notifications were off in my git profile...)

Am I correct in understanding that gorhill experienced exactly what I was dealing with? The UI resetting and 3P filters needing reloading?

Still resetting the UI but at least 3P filters are stable and retained.

arkenfox / user.js