uMatrix WE problem in combination with other WEs [Bug 1417249, 1477696, 1421725...]

[earthlng]

@gorhill, I found a problem with uMatrix in that, depending on the order in which FF loads the addons, uMatrix "restores" the Content-Security-Policy header when another addon already changed it or removed it completely. Maybe other headers are affected as well, IDK. uBO doesn't have the same problem so I assume you already fixed it there.

Since you closed your Issue tracker for uMatrix I didn't know how else to contact you. If you're interested let me know, I have a simple example addon ready to share and the steps to reproduce. Thanks

[gorhill]

uMatrix "restores" the Content-Security-Policy header

Were 1st-party scripts blocked?

[earthlng]

Yes but I just re-tested with 1st-party scripts allowed and it's the same.

[earthlng]

tested in FF nightly 58 with uMatrix 1.0.1rc3

example addon: manifest.json

{
  "name": "demo",
  "version": "0.1",
  "manifest_version": 2,
  "background": {
    "scripts": ["background.js"]
  },
  "permissions": [
    "<all_urls>",
    "webRequest",
    "webRequestBlocking"
  ],
  "applications": {
    "gecko": { "id": "demo@demo.org" }
  }
}

background.js:

function logResponse(e) {
  for (let header of e.responseHeaders) {
    if (header.name.toLowerCase() === "content-security-policy" || header.name.toLowerCase() === "test-test") {
      console.log(e.url + "\n" + header.name+":", header.value);
    }
  }
}
function editResponseHdrs(e) {
  e.responseHeaders.push({'name': 'Test-Test', 'value': 'Test123'});
  e.responseHeaders.push({'name': 'content-security-policy', 'value': ''});
  return {responseHeaders: e.responseHeaders};
}
browser.webRequest.onHeadersReceived.addListener(editResponseHdrs, { urls: ["http://*/*", "https://*/*"] }, [ "blocking", "responseHeaders" ]);
browser.webRequest.onResponseStarted.addListener(logResponse, { urls: ["http://*/*", "https://*/*"] }, [ "responseHeaders" ]);

example testpage: https://gist.github.com/arantius/f6fd80b1efad368a45ca35567bc31b18

steps to replicate:

• install uMatrix, create the 2 files and load it as a temporary addon, open the console and visit the testpage
  • result: the test-header is output on console and the CSP is not => the demo addon removed the CSP header and I assume uMatrix sees that and doesn't need to append its special rule to it.
• disable and re-enable uMatrix in about:addons then reload the testpage
  • CSP header output on the console --> somehow the demo addon's modifications are ignored and uMatrix still knows the original CSP header
• clear the console, reload the demo addon in about:debugging then reload the testpage
  • CSP is now cleared again by the addon

I added the test header because I was wondering if uMatrix overwrites all modifications another addon made but it seems to only affect the CSP one.

At first I thought it might be because in my original addon I process the headers async but when I changed it to sync it didn't fix the problem. I also think CSP is maybe not the only header where this could be a problem. It particularly sucks because when I start Firefox it seems to always load my addon first and uMatrix always wins and I need to disable + re-enable my own addon for it to work as expected.

[gorhill]

I am not seeing anything wrong with uMatrix -- it append its own script-src unsafe-eval * only when first party scripts are blocked, so working as expected.

However the behavior of the webRequest API does not match the documentation, specifically:

If two extensions listen to onHeadersReceived for the same request, then the second listener will see modifications made by the first listener, and will be able to undo any changes made by the first listener.

uMatrix sees the headers without your changes, and your extension sees the headers without uMatrix changes -- depending on whichever listener was first called.

[earthlng]

Yeah I guess that would explain what's happening. I assumed that mozilla's documentation is accurate but apparently not. If the behavior matched the documentation it wouldn't be a problem then, right? Sorry for blaming uMatrix when in fact it seems to be a FF problem ;) Can you inform them about it, please? I think they're more likely to fix it when you're the one reporting it.

[Thorin-Oakenpants]

I have read a few times about the order of installation causing issues. Earthlng - have you tried installing uM and yur Ext in a TWO new profiles in the two orders possible to see if that makes a difference to which Ext beats the other?

If that works for you, then try uninstalling (and removing all traces) and reinstalling the applicable extension (edit: in your main everyday profile) - not sure if it will work as FF often remembers stuff - I remember a ticket gorhill was concerned about where something has a hook on it - I think it was uBo prefetching/hyperlink/WebRTC settings or something.

[earthlng]

Even if that would work, every new addon update would probably fuck things up again. They need to fix the behavior to match the documentation otherwise it's a complete mess and totally unpredictable and unreliable.

[earthlng]

Every addon update reloads the background script(s) which re-registers the listeners, meaning the addon that just got an update ends up being the one that has the last word now, so to speak. For example addon1 appends a cookie and a 2nd addon adds another cookie but since the 2nd addon doesn't see the modified cookie header from the 1st addon, the 1st addon's cookie is never set.

Edit: kind of a bad example because there's a special API to deal with cookies but it can also be done on the request level by modifying the cookie header and in that case it could be a problem.

[earthlng]

https://bugzilla.mozilla.org/show_bug.cgi?id=1417249

[Atavic]

If possible, only one extension should change a specific header to avoid any problem. https://github.com/pyllyukko/user.js/issues/348

[earthlng]

Looks like they're not gonna fix this and instead will just change the documentation: https://bugzilla.mozilla.org/show_bug.cgi?id=1421725#c1

That means it will always be like rolling a dice, and every time you start FF you'll roll the dice again. Expect the unexpected if, for example, NoScript, uBlock Origin (in advanced mode!) and uMatrix are all used together because afaik they all work by changing the CSP header.

Roll the dice and hope for the best, I guess :man_facepalming:

[gorhill]

At least Chromium reports conflicts to users, that's the least Firefox could do to bring awareness of possible incompatibility between extensions. Imagine an extension which relax CSP directives[1] after NoScript/uMatrix made them more strict -- surely a user would want to know a security/privacy extension is prevented from working properly because of some other extensions.

Comment:

adding cookies is not really a good use case for this

Sure, but extensions changing CSP header is quite a serious use case.

---

[1] I think Tampermonkey does this.

[gorhill]

Is this for cookies and headers and other stuff?

It's for request/response headers. "Cookies" is mentioned because the case brought forth was about the Set-Cookie header.

[gorhill]

Side note: I get a LOT of this crap: what does it all mean

Difficult to tell exactly, I would have to know exact URL + extensions and their configuration. It seems some in there are caused by uBO redirecting Google Analytics script to its neutered version.

[earthlng]

And once again the argument is performance. I'd think users would prefer that their addons work as expected but what do I know.

[Thorin-Oakenpants]

And once again the argument is performance. I'd think users would prefer that their addons work as expected but what do I know.

How do you two guys feel about putting this out in front via a ghacks article. Seems pretty serious to me

Edit: put some pressure on? We'd need a decent example of security/privacy failure or extension failure

[gorhill]

I only get a warning on my side with uBO + uMatrix:

Content Security Policy: Directive ‘frame-src’ has been deprecated. Please use directive ‘child-src’ instead.

With uBO + uMatrix, and this one is because of the "Forbid web workers" setting in uMatrix (warning occurs whether workers are blocked or not). I don't know where the other errors/warnings come from.

[earthlng]

@gorhill you could setTimeout + re-register the listeners in the background script of uMatrix to somewhat make sure it's always the winner but that would only work as long as other addons don't start to do that as well. It would also not guarantee that uMatrix keeps having the last word when new updates of other addons are installed. You could do it in intervals but that would only make things even worse if others start doing the same thing, because it would continuously roll the dice during a session and not just on startup.

uBO with only a * * * noop rule should work just fine with uMatrix though, right? Do no-remote-fonts rules in uBo work by blocking the request based on URL and request-type, or is that CSP-based too?

[Thorin-Oakenpants]

^^ its Violent Monkey, which I only use for our three global scripts in the wiki

[gorhill]

uBO with only a * * * noop rule should work just fine with uMatrix though, right?

Well, a * * * noop is meaningless in uBO, there is no block/allow rules to override at * * * level. If you mean that without using dynamic filtering in uBO, there should be no clashing, it's not entirely the case: uBO injects a font-src CSP when "No remote fonts" is checked. Without such block switch however, yes, there should be no clashing.

[earthlng]

Pants, if you want to test then check what gorhill alluded to regarding Tampermonkey. If what he suspects is correct that would mean TM could completely override uMatrix' settings.

But please don't take this to ghacks! I still have enough faith in the mozilla devs to think that they'll do something about it when @gorhill is the one telling them how bad this is.

[earthlng]

Well, a * * * noop is meaningless in uBO

well now I feel stupid, thanks xD I think I added that rule when I noticed the about:addons page making requests to google-analytics and uBO didn't block it. So I removed everything in the whitelist and replaced the 2 default BTS rules with that single noop-everything rule. Does that make some kind of sense at least? ;)

[earthlng]

@gorhill I found a site that illustrates the problem (or at least 1 problem): https://1337x.to/home/

Tested with the latest Firefox Nightly and the latest uBlock + uMatrix versions

CSP when uBlock comes after uMatrix (= disable + re-enable uBlock to achieve that) : Content-Security-Policy: default-src 'self' * data: 'unsafe-inline' 'unsafe-eval' -> result: site loads

umatrix comes last (= disable + re-enable uMatrix) : Content-Security-Policy: script-src 'unsafe-eval' blob: * -> result: site content is hidden and only the background image is visible

I've already disabled "i'm an advanced user" which I thought might have something to do with it, and I also disabled font-blocking and CSP-reports blocking, for the same reason. But apparently the csp comes from this blob filter in easylist: |blob:$domain=101greatgoals.com|1337x.to ... ... and overwrites uMatrix's more restrictive rule.

Questions:

1. Is there a way to completely prevent uBlock from changing the CSP header ie ignore these blob filters?
2. Are there other filter types that can cause something similar?
3. As for csp-report blocking and given that the header-processing across multiple extension is not very reliable, wouldn't it be better to block CSP-reports on the request-level with the csp_report ResourceType ? Or is that not reliable enough in your opinion? There's also the font ResourceType which, if blocked that way, could allow users to block both Fonts and CSP-reports in uBO without the risk of overwriting uMatrix' settings.

And if you don't mind, can you explain why uMatrix is using unsafe-eval instead of none, and what the blob: * is for? Thanks

[gorhill]

Is there a way to completely prevent uBlock from changing the CSP header ie ignore these blob filters?

No. I will add an advanced setting in uBO to disable completely CSP header injection/modification.

Are there other filter types that can cause something similar?

Yes, filters with csp= option (currently quite rare).

As for csp-report blocking

No issue here. uBO blocks the csp-report network requests, this is unrelated to the modification of response header. uMatrix injects a CSP-Report (with an invalid URL) only so that a SecurityPolicyViolation event is triggered in the DOM.

can you explain why uMatrix is using unsafe-eval instead of none

Because the rule related to that CSP directive is to block use of inline-script tags, not eval(), which could be used legitimately in some non-blocked (1st- or 3rd-party) scripts.

what the blob: * is for?

To be able to enforce the filters such as |blob: as seen in EasyList. Blob-based URLs do not go through the webRequest API, and thus the only way to block them is through a CSP directive.

[earthlng]

No. I will add an advanced setting in uBO to disable completely CSP header injection/modification.

@gorhill have you had the time to implement this yet? I looked through all your release notes but couldn't find anything related.

uMatrix injects a CSP-Report (with an invalid URL) only so that a SecurityPolicyViolation event is triggered in the DOM.

I assume this is used to detect if workers are used, right? Anything else?

Firefox recently added a new pref security.csp.enable_violation_events but AFAIK it still defaults to false ie violation events are disabled - I guess that means uMatrix works with or without this pref enabled?

[gorhill]

Sorry, not implemented.

Yes, uMatrix uses security policy violation events to detect workers. I didn't know about security.csp.enable_violation_events -- I checked and it looks like this is breaking uMatrix's ability to detect workers.

Edit: related: https://bugzilla.mozilla.org/show_bug.cgi?id=1425993

[Thorin-Oakenpants]

ummm .. so we set security.csp.enable_violation_events=false in the user.js - does this mean uM's no-workers rules no longer work?

[gorhill]

The rule works, the detection of workers won't work. worker won't be reported in the logger, and the dot in the switch knob won't be there when there are workers.

[earthlng]

@gorhill

Sorry, not implemented.

no problem. Are you still planning to implement it or have you decided against it?

It really sucks that they refuse to fix this, especially since their only argument is performance!

@fmarier could you maybe take this up internally to get fixed, please? I'd imagine you agree that sacrificing extension inter-compatibility, functionality and consistency for a slight performance improvement is not ideal, to say the least. I'm pretty sure most users with 2 or more potentially interfering extensions would be more than happy to sacrifice a little bit of a performance for the guarantee that their extensions work nicely together. Atm it's totally unreliable and unpredictable and you might get one outcome once and a completely different outcome the next time (like after an update for one extension or after a restart). Even the guys in charge at mozilla say it's like rolling a dice! ...

• "If multiple listeners try to change the same header, one of them will succeed, but there's no guarantee which."
• "its rolling the dice to see if you'll win."
• "The entire header change API is always a game of rolling dice."

It's a sad state of affairs for a privacy-respecting browser like FF when it makes using privacy and security enhancing extension like NoScript, uBlock Origin, uMatrix, etc "a game of rolling dice" IMHO

[gorhill]

have you decided against it?

I forgot about it, and really it's just a time thing in the end. This project is still a hobby and I try to stick to what solve actual issues for most users coupled with what I feel like doing at the moment -- there is also real life stuff to do too in the end.

Edit: To be clear, any issue should be formally reported on the issue tracker, or else I will just forget about it. When I feel like working on the project with no specific idea of what to do, I visit the issue tracker.

[earthlng]

no problem, I totally understand and I wasn't trying to blame you or anything. Was just wondering if it's still in the cards or whether I'll have to edit my own copy once I upgrade from ESR52.

github tells me "An owner of this repository has limited the ability to open an issue to users that have contributed to this repository in the past." when I try to open a new issue in your uBlock repo.

[Thorin-Oakenpants]

uBO and uM have new issue repos

• https://github.com/uBlockOrigin/uBlock-issues
• https://github.com/uBlockOrigin/uMatrix-issues

I'm guessing this was to give gorhill some breathing space / respite and allows others like ublockuser, jspenguin, gawser etc to handle more stuff

[earthlng]

thanks, I didn't know that. I was looking at https://github.com/gorhill/uBlock/issues

[Thorin-Oakenpants]

https://bugzilla.mozilla.org/show_bug.cgi?id=1417249 - closed marked duplicate of this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1477696

Edit: also gained some traction: P2 for now

[gwarser]

Maybe related: headers are merged https://bugzilla.mozilla.org/show_bug.cgi?id=1377689 (fixed) but only when present in original response https://bugzilla.mozilla.org/show_bug.cgi?id=1462989

[claustromaniac]

However the behavior of the webRequest API does not match the documentation, specifically:

If two extensions listen to onHeadersReceived for the same request, then the second listener will see modifications made by the first listener, and will be able to undo any changes made by the first listener.

uMatrix sees the headers without your changes, and your extension sees the headers without uMatrix changes -- depending on whichever listener was first called.

Well, shit. Why wasn't I aware of this?!

Does this happen too with OnBeforeSendHeaders handlers (for modifying request headers)? AFAICT it doesn't, but I'm not sure.

[Thorin-Oakenpants]

:cat2: - just assume the worst. I keep reading comments like "update the documentation" rather than "fix the problem"

[claustromaniac]

That's... tragic.

[earthlng]

let the listener wars begin ... https://github.com/hackademix/noscript/issues/6

[Thorin-Oakenpants]

FYI: https://bugzilla.mozilla.org/show_bug.cgi?id=1485526 Consider allowing users to specify the order in which extensions' webRequest listeners are run

Edit: and I guess that's not feasible .. WONTFIX

[earthlng]

It's very unlikely that mozilla are ever going to fix this. This issue has been open for over a year and people should be aware by now.

[tartpvule]

Just want to share a dirty and hacky patch for Bug 1421725 I made that makes the behavior matches the documentation on MDN: https://github.com/tartpvule/my-firefox-patches I am hesitant to propose this on Bugzilla given the direction of that bug discussion toward changing the documentation, though.

Cheers

[earthlng]

Holy shit @tartpvule, you went deep there! Very impressive and very interesting, thanks a lot for sharing that! I'm a bit hesitant to try it though because without debugging your code to an insane degree I have no idea about what the potential side effects might be or how well your code actually works. And since I know pretty well what all my installed addons do, I'm not that affected by the problem because I work around it with a custom version of uBlock Origin. Very impressive work none the less and a very nice example of the extent of what's possible with autoconfigs. Thanks again for sharing that, it gave me a few ideas for some other potentially neat little tweaks.

btw, I tried your other script for allowing unsigned extensions but it doesn't work in 65 anymore. An updated version of the method described at https://www.ghacks.net/2016/08/14/override-firefox-add-on-signing-requirement/ at least allowed me to install unsigned extensions but I haven't tested if those extension are loaded successfully on startup, etc. Anyhow, I switched to dev-edition for the sole reason that it allows me to run unsigned extensions, which IMO is the easier solution

cheers

[Thorin-Oakenpants]

While we're at it, @gwarser said HTTP Everywhere also clashes with uBO etc - I read it on reddit - can't find it now. @gwarser : is this true?

[gwarser]

If I'm reading it right, HTTPS Everywhere uses upgrade-insecure-requests CSP directive when option (?) to block all insecure requests is enabled (httpNowhereOn)

https://robwu.nl/crxviewer/?crx=https%3A%2F%2Faddons.mozilla.org%2Ffirefox%2Fdownloads%2Flatest%2Fhttps-everywhere%2Fplatform%3A2%2Fhttps-everywhere.xpi&q=!content-security-policy&qf=background-scripts%2Fbackground.js&qb=1&qh=1&qi=1

[Thorin-Oakenpants]

oh: https://github.com/ghacksuserjs/ghacks-user.js/issues/265#issuecomment-472024841

OK .. ^^ one post up then since its duplicated now :)

when option (?) to block all insecure requests is enabled (httpNowhereOn)

I wouldn't mind sorting this out. I don't block all insecure requests (I rely on my speed dials and shit, and to not allowed any mixed content) .. but if the above is true then we should add that info to the wiki extensions (just like we did with Canvas Blocker's one setting)

Edit: PS: thanks @gwarser :kiss:

[Thorin-Oakenpants]

re-opening to track HTTPS-E

[tartpvule]

@earthlng Thank you for the comment

1. Due to https://bugzilla.mozilla.org/show_bug.cgi?id=1455601, AutoConfig was sandboxed and thus prevented from monkey patching modules starting from Firefox 62, but ESR users can set general.config.sandbox_enabled to false in a prefs.js (the same file you set general.config.filename) to disable this sandboxing. So the targets for AutoConfig-based patches are ESR only; they're probably not going to work on release builds.
2. I am personally using Bug1421725_WebRequest.patch in my private ESR build on a Linux machine, and monkey_Bug1421725_WebRequest.cfg on my Windows machine running official 60.5.2esr. They work :)

I really hope Mozilla actually fixes WebRequest to do what MDN says and not the other way around. As it stands, only one random extension can modify a header, so uBlock cannot be used alongside uMatrix reliably; that's bad.

[gwarser]

HTTPS-E has an option in the dropdown to use "Encrypt All Sites Eligible (EASE)" which is default off. Is this the options that turns on the CSP?

I'm pretty sure this is it.

Makes me wonder what other (of our recommended) extensions are using CSP

Open them in CRX viewer and search for !content-security-policy. (! means scan all files.) If they touch this header, they are potentially unsafe.

[earthlng]

@tartpvule yes I know about the sandbox and disabled it during my testing. So, atm FF65+ still allows to disable the autoconfig sandbox. 100% agree with the rest of your comment.