search

uBlockOrigin / uMatrix-issues

This is the community-maintained issue tracker for uMatrix
https://github.com/gorhill/uMatrix
123 stars 17 forks source link

All pages take forever to load #269

Open homo-programmatis opened 4 years ago

homo-programmatis commented 4 years ago

Prerequisites

Description

After Firefox's storage becomes damaged, uMatrix causes all pages to take forever to load.

If Firefox is to blame, I would still suggest to handle the error gracefully, rather then just silently blocking all pages from being loaded (which renders entire Firefox unusable until problem is fixed). Probably show a reasonable error message to help user get over it?

The problem has happened to me 3 times now: today, a couple weeks ago and around a year ago.

I understand that other users faced this too, for example #264 - and I think I saw others, but can't find them now.

A specific URL where the issue occurs

Any url.

Steps to Reproduce

  1. Close your Firefox.
  2. Go to your Firefox <profile_name>\storage\default. For example, on Windows: %appdata%\Mozilla\Firefox\Profiles\<profile_name>\storage\default
  3. Backup, then delete all files in this folder. Can't say why this is important.
  4. Extract attached archive: 7.zip. I understand that it contains a damaged storage entry. Can't say what exactly is wrong about it.
  5. Start Firefox, try to navigate to any URL. Firefox will take forever to load the page.

Ruleset

Unrelated to ruleset.

Supporting evidence

Tools | Web Developer | Browser Console will have this:

20:22:58.060 IndexedDB UnknownErr: ActorsParent.cpp:570
20:22:58.064 UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code. ExtensionStorageIDB.jsm:831
    normalizeStorageError resource://gre/modules/ExtensionStorageIDB.jsm:831
    method chrome://extensions/content/child/ext-storage.js:273
    AsyncFunctionThrow self-hosted:697
20:22:58.168 IndexedDB UnknownErr: ActorsParent.cpp:570 2
20:22:58.274 UnknownError: The operation failed for reasons unrelated to the database itself and not covered by any other error code. ExtensionStorageIDB.jsm:831
    normalizeStorageError resource://gre/modules/ExtensionStorageIDB.jsm:831
    method chrome://extensions/content/child/ext-storage.js:273
    AsyncFunctionThrow self-hosted:697
20:22:58.274 µm.userSettings is undefined storage.js:136
    settingsLoaded moz-extension://d7631edc-38b5-4fa6-85c4-c4a8d89027b6/js/storage.js:136
    apply self-hosted:1870
    applySafeWithoutClone resource://gre/modules/ExtensionCommon.jsm:614
    wrapPromise resource://gre/modules/ExtensionCommon.jsm:843
    withLastError resource://gre/modules/ExtensionCommon.jsm:762
    wrapPromise resource://gre/modules/ExtensionCommon.jsm:831

Your environment

gorhill commented 4 years ago

Please see if you can also reproduce with dev build: https://github.com/gorhill/uMatrix/releases/tag/1.4.1b6

homo-programmatis commented 4 years ago

Yes, reproducible with 1.4.1.6. In case you didn't notice: I have provided reproduction steps. I have verified that they work when starting with a clean profile.

gorhill commented 4 years ago

In case you didn't notice: I have provided reproduction steps.

Can you share why you felt the need to say this?

homo-programmatis commented 4 years ago

Totally no offense meant. The report page (especially with header) is quite long. In other couple reports I glanced at, reproduction steps are not really reproduction steps. So I imagined that it's easy enough to not notice that this time, reproduction steps actually do reproduce.

uBlock-user commented 4 years ago

I would still suggest to handle the error gracefully

It's not just an error, storage component is broken and that breaks uMatrix along the way along with all extensions too relying on that component.

gorhill commented 4 years ago

I would still like to see if the start code can better deal with such error -- now that I have a test case I will be able to step in the code as the error occurs.

homo-programmatis commented 4 years ago

Let me try to provide some reasoning.

Such cases inevitably fall in some border zone, where some other component (Firefox, in this case) can be blamed and that's it, discussion over. I agree, my naive understanding is that Firefox should have dealt with the problem without letting it cause more damage.

However, let's see it with the eyes of an average user:

  1. Firefox gets broken almost to 100% extent where it stops loading any pages, which is its primary function.
  2. It's super hard to figure what to do to get Firefox repaired.
  3. Possible outcome A: user can't solve it and abandons Firefox for good. Sounds like a pain to me.
  4. Possible outcome B: user can't solve it and reinstalls OS. This is already a pain. Then either the entire profile is discarded (another pain: will need to redo all bookmarks passwords etc); or user tries to reuse profile and gets the problem back (problem not solved).
  5. Possible outcome C: user finds that disabling uMatrix works; uMatrix gets blamed for breaking Firefox; uMatrix is deleted and maybe advertised as a bad thing. From user's perspective, that's a totally valid conclusion. That would make me sad: I like uMatrix!
  6. From my naive understanding, problem is rather easy to detect programmatically. If true, that would mean that cost of implementing is low, benefit for users is high. Even a suggestion like "delete entire storage" is much better for users compared to (3)(4).
succeedmr123 commented 4 years ago

Dear and appreciated @gorhill & @homo-programmatis who disclosed this very unpleasant issue first,

I can fully second the observations stated here, all points 1-6 nails the user choices. It all happened to me with the another addon. Addon data is still inaccessible in its own IndexDB, addon does not function at all (as it can not acces data), and it happened now with uMatrix again.

I will try to get to the suspected culprit as short as possible, and propose a minimal FF auto-check fix mechanism;

In my opinion, we have a case of Firefox not having a resilient mechanism of protecting IndexDB data structures, these are confirmed by my investigation at least. Simply, the data in an IndexDB are hanging there to be corrupted at the mercy of a sudden power loss or any similar disaster, without autofix chackup.

Last time it happened (and is not addressed asof yet at all) was the addon called "Tab Session Manager".

So, suspected cause of this is similar to IndexDB corruption.This time did not have the moment yet (as I deal with 16 hours production time here). Or, from my surface user level observation, a general issue of unfixed.. any kind of a setup, database, config file, if a sudden power loss occurs.

I found myself dumbfounded with Firefox/addon weird behaviour now for the second time - this time with uMatrix. However, it seems it is not uMatrix direct "fault". Firefox local data structure for the addon is, I speculate, broken, and any pages totally refuse to load due to uMatrix not being able to load its data, and then not letting pages get the datastream. Under uMatrix settings, I see 0 bytes in local storage, talking from an user superficial overview.

Given my experience with the first addon with broken IndexDB "Tab Session Manager", something like that is the culprit here as well.

Firefox, as any complex system of OS calibre of complexity, should be having a mechanism to autodetect dirty data and before start - autorepair it, either from backup, or live.

Can not be normal that the users get into a situation of more than one addon get Firefox stalled, or not functioning, because the addon can not load data that is not sanitized properly.

And users themselves not being able to do anything about it.

For instance, how I found at all that IndexDB was at fault;

  1. During normal ops just observed addon has no data ("Tab Session Manager" in this case)
  2. During interenet search on how to recover my user data, found just a vague reports of IndexDB broken in Firefox (Nothing helped.. safe modes etc..)
  3. Identified data for the "Tab Session Manager", and found that indeed there was an IndexDB folder for that addon

was loking for IndexDB fixer out at the internet - no dice.

So, average user, even the one who have the best willpower to leave production machine workflow and investigate, can not do it without the platform mechanism /fix for similar data corruption scenarios

Now, I have another situation with a broken addon, and its uMatrix. It's off for the third day (icon itself is all greyed out, not functioning), and I am forced to be running sites with privacy concerns, my (i.e. user's) data up to the date of breakage is inaccessible yet, without any way to fix the situation for now. Just a first-hand user experience.

I am almost ready to jump in fire and start debugging hard-core out of sheer feeling of frustration at the inadequacy of the current FF robustness, either desktop release, or Preview Edition for Android, who started to crash 90% of the time.

Best,

succeedmr123 commented 4 years ago

Dear respected @gorhill and @homo-programmatis

two additional QA tests are documented here in an effort to exclude the culprits til eventually pinpointing the exact factors contributing to the issue;

First crude / fast glance checkup reveals this: (TL/DR, different machines work differently it seems windows 10 v1909 64 bit has to do with it in some way according to my preliminary tests)->

Platform Test: uMatrix on few other computers - different systems, same Firefox account, logged in, and syncing data.

Functional test - lets get to results:

PC1: Win 10 v1909 64 bit, Firefox 64 bit of course, synced, uMatrix addon on: Result: unusable, traffic blocked, click on the grey uMatrix icon gives No web page, needed to shut the addon off.

PC2 Win 10 v1909 64 bit - also unusable, addon also need to be shut off in order for Firefox to load any page at all

PC3 Win7 32 bit, Firefox ESR 32 bit - it works well

Hope the obvious results may help.

Test 2: Test by disabling ASLR anti-exploit under Windows defender options per-app for firefox as exclusion (TL/DR, attempt 1, test did not did not make any difference)

Turned off Antivir exploit "obvious suspect" - ASLR page anti-exploit technique 1 and 2 (there are two methods near the end of the list within windows defender)

Later will proceed with excluding Firefox for all security anti-hack methods stated there in order to observe the behaviour (We have known confirmed test that e.g. Jetbrains IDE get totally messed up by Defender anti-exploit tech)

Hope that helps

homo-programmatis commented 4 years ago

This is how to fix and find which folder is causing the problem:

  1. Go to your Firefox <profile_name>\storage\default. For example, on Windows: %appdata%\Mozilla\Firefox\Profiles\<profile_name>\storage\default
  2. Backup the entire folder.
  3. Delete half of the folders with names that begin with https+++.
  4. If Firefox is still broken, make another backup (but keep the previous backups).
  5. If Firefox works, delete entire folder and restore from last backup.
  6. Repeat 3-5 until you only have 1 folder that causes problem.
  7. Restore everything from the first backup.
  8. Delete the offending folder.
  9. You fixed your Firefox.
succeedmr123 commented 4 years ago

@homo-programmatis

I appreciate the basic logic flow outline, however..

Reasons not to do that way, the alternative, and a long-term fix:

  1. Way too time-expensive
  2. You would destroy your own data in the proposed way, which if you have a long-time built database with your own private data- is simply unacceptable.
  3. Firefox workaround already works (just turn off the addons with still-unfixed data), albeit without the addons whose data is still invalid.

ALTERNATIVE to fix the unacceptably long time expense - its way more easy to find addon ID, then matching with addon Folder with the similar ID, and if anyone is into destroying his own data, click delete and reinstall the addon

Long term fix: autocheck at addon, or Firefox level, as a system to autofix the misbehaving databases, and anything inbetween. Think something (metaphorically) along the lines of M$ Chkdisk, fixboot,

rossengeorgiev commented 4 years ago

Hi, I've been encountering this issue since March. Basically, storage breaks somehow when a site uses it. Say, youtube does it. Open YouTube, close browser, re-open, nothing loads until site folders are deleted from storage/default folder. Here are the errors from the console:

image

I not able to replicate the behaviour on another PC. I wonder if there is feature flag, or settings, in the browser that could be causing the different behaviour?

gwarser commented 4 years ago

Check storage/default for very long file/folder names, unknown files (also hidden), trailing dots in file names. Try toggling dom.quotaManager.useDOSDevicePathSyntax = false or network.file.disable_unc_paths = false

Recently this was the problem https://bugzilla.mozilla.org/show_bug.cgi?id=1536796 but Firefox storage/quota manager is full of issues.

rossengeorgiev commented 4 years ago

I have no trailing dot, and the paths are <130 chars. Toggling the suggested settings had no effect.

image

SyntevoAlex commented 4 years ago

If you have some patience, please try my algorithm from https://github.com/uBlockOrigin/uMatrix-issues/issues/269#issuecomment-646347807

If you already suspect that youtube causes it, could you please backup&remove all other folders to verify that just the youtube's folder is enough?

rossengeorgiev commented 4 years ago

hi @SyntevoAlex, here is a video to illustrate. This is the minimum repro case I have https://streamable.com/voi74a This is with umatrix disabled, so looks like Firefox bug.

Raised the issue here: https://bugzilla.mozilla.org/show_bug.cgi?id=1647323

homo-programmatis commented 4 years ago

Thanks for the video. Your case looks pretty bad indeed. You did the right thing reporting it to Mozilla directly. In my case, the issue is not easily reproducible.