tracking issue: WebExtension port (aka. Firefox Quantum / Firefox 57+ support)

[myrdd]

Starting with Firefox 57+, only WebExtension Add-Ons will run in the browser. RequestPolicy needs to be ported to the WebExtension API.

You want to help with this transition? The first thing you can do is to install the “Development Channel” version [1] of RequestPolicy Continued and to report any bug you encounter with this specific channel. Thank you.

[1] RPC dev-channel: ~on the bottom of page https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/~ [Edit, 2018-04-04] https://sslsites.de/requestpolicy.256k.de/requestpolicy-nightly.xpi [Edit, 2018-10-25] https://requestpolicy.256k.de/requestpolicy-nightly.xpi

[acrobat1]

In layman's terms, what would this mean for RP ? Will it continue to work, or require major rewriting to do so ? Can we users expect the same level of functionality, or will it be somehow restricted by this change? Thanks.

[ivan-kolmychek]

@myrdd I agree, some explaination would be just wonderful, because not everyone has time and experience to get through all the docs and understand all the consequences.

In WebRequest documentation I've not seen anything that would actually prevent the RP from blocking a request, but I don't know how much of the requests will actually be exposed through it - for instance, can we block requests of plugins?

I'm also concerned about some details of Chrome's implementation:

Moreover, only the following schemes are accessible: http://, https://, ftp://, file://, or chrome-extension://. In addition, even certain requests with URLs using one of the above schemes are hidden, e.g., chrome-extension://other_extension_id where other_extension_id is not the ID of the extension to handle the request, https://www.google.com/chrome, and others (this list is not complete).

I've not found anything about that in firefox wiki, but I've not looked for it too much yet, just read the Chrome's verstion.

[myrdd]

TL;DR / Very short answer: There's no need to believe RP is going to discontinue to work. Some major rewriting might be necessary in the long term. I will make sure that RP won't lose functionality.

---

Mozilla's aim of the announcement was to start discussion with the community. The discussion is about the WebExtensions API, a „new browser extension API“.

Right now the technologies used to create extensions are XUL and XPCOM. Creating an API doesn't mean that the old technologies are dropped.

Still, we are not sure if XUL/XPCOM is dropped in a few years or not, so I think it's important that the API provides access to all functionality needed by add-on developers. I believe that someday Mozilla's future Layout Engine Servo, which is still experimental, will replace the current Layout Engine Gecko, which is used in Firefox and SeaMonkey. Servo is completely rewritten, and afaict it does not implement XPCOM. Now, what I believe is that Mozilla wants to develop the WebExtensions API independently from the underlying technology so that it can be supported by both Gecko and Servo extensions.

In my opinion Servo is a great project, but also the programming language it's written in, Rust, which has been developed by Mozilla Research as well. Having a browser with Servo/Rust someday in the future will be a great effort.

Now, what does this mean for RequestPolicy? It means we need to make sure that anything needed by RP will be supported by the WebExtensions API.

The XUL technology is not that important to RequestPolicy. However, the XPCOM technology is. One very important part of XPCOM used by RP is nsIContentPolicy.shouldLoad. The corresponding API to that function is WebRequest.onBeforeRequest. Still, the API isn't equivalent to shouldLoad yet. The parameters I need to request are definitely aRequestOrigin (the origin URI) and aContext (e.g. the HTMLElement in the DOM); maybe also aContentType and aRequestPrincipal.

[myrdd]

I've updated the first comment to represent my current knowledge. Input and investigation welcome.

[myrdd]

Just to give an update, I'm currently working on a WebExtension version of RPC.

[myrdd]

All those willing to help me with the transition to WebExtensions, please install the “Development Channel” version [1] of RPC, and report any bug you encounter. Thank you.

[1] on the bottom of page https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/

[genodeftest]

@myrdd Is it intended that Firefox still takes the extension as not-e10s-compatible?

[myrdd]

@genodeftest yes, RPC still isn't e10s-compatible. compatibility will come with the migration to webext.

[Pentarctagon]

Is it correct that the version in the dev channel is 1.0.beta12.4, while the currently released version is 1.0.beta13.0?

[myrdd]

This wasn't intended. Seems like I misread the documentation, so release/beta channels are really separated: users on the dev channel won't get an update if a new release version is released, even if the version number and release date are newer. However, the only difference between the newest dev-channel version and the beta13 is the version number (and a fixed UI test), as shows the diff. Thanks for the notice anyway.

[DJ-Leith]

@myrdd I hope you can port / rewrite RPC for Firefox 57.

Input and investigation welcome.

In my opinion, a good place to start reading is Aris (who created "Classic Theme Restorer") clear explanation here:

"Legacy add-ons like CTR will stop working when Mozillas XUL/XPCOM support ends with Firefox 57 release" https://github.com/Aris-t2/ClassicThemeRestorer/issues/299

There are several very helpful links there.

In addition to the links in Aris' article see also these 3:

"Add-ons/Firefox57" https://wiki.mozilla.org/Add-ons/Firefox57 This is Mozilla's Official wiki (but as it is a wiki it changes, so keep checking).

Andy McKay, who is very influential in the move to WebExtensions, has posted on some Official Mozilla blogs (you can search for them).

This post, from his personal blog, gives a clear insight as to WHY he is so keen on WebExtensions. http://www.mckay.pub/2016-11-26-innovation/

"Stuff we can remove when XPCOM extensions are no longer supported" https://bugzilla.mozilla.org/show_bug.cgi?id=1347507&format=default See the "Depends on:" field, there are dozens of Bugs there.

As I am cautious, I moved all my important Profiles, in April 2017 (while Firefox Release was version 52), to the ESR 52.x.x

Nightly is already at version 57 and I expect 'things to break'.

DJ-Leith

[myrdd]

I've released a new pre-version on the development channel. It introduces many code changes necessary for WE migration. The pre-version does not contain new features, and it's not a WE yet. Please install and test the pre-version: https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/versions/beta

[genodeftest]

@myrdd : I'm running your pre-version for about a week without visible functionality issues.

On every start, I get these lines printed to console:

console.debug: 
  [RequestPolicy] starting up
console.info: [RequestPolicy] [POLICY] PolicyManager::loadUserRules loading user rules
console.info: [RequestPolicy] [POLICY] PolicyManager::loadSubscriptionRules: official / allow_extensions
console.info: [RequestPolicy] [POLICY] PolicyManager::loadSubscriptionRules: official / allow_sameorg
console.info: [RequestPolicy] [POLICY] PolicyManager::loadSubscriptionRules: official / allow_mozilla
console.info: [RequestPolicy] Will update subscription: official allow_extensions
console.info: [RequestPolicy] Will update subscription: official allow_sameorg
console.info: [RequestPolicy] Will update subscription: official allow_mozilla
console.info: [RequestPolicy] Will update list: official
console.info: [RequestPolicy] Updating [SubscriptionList official https://raw.githubusercontent.com/RequestPolicyContinued/subscriptions/master/official.json]

Also, since today I'm getting this error message printed to console/syslog:

console.error:
  [RequestPolicy] Fatal Error:
console.dir:
  Message: [Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIURI.asciiHost]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://rpcontinued/content/lib/request.js :: Request.prototype.isInternal :: line 76"  data: no]
  Stack:
    Request.prototype.isInternal@chrome://rpcontinued/content/lib/request.js:76:1
NormalRequest.prototype.isInternal@chrome://rpcontinued/content/lib/request.js:160:14
self.process@chrome://rpcontinued/content/lib/request-processor.js:262:17
self.shouldLoad@chrome://rpcontinued/content/main/content-policy.js:60:16

This happened 5 times today, but I cannot tell you when it happened, i.e. I don't have a reproducer.

[myrdd]

@genodeftest which browser and version are you using?

[myrdd]

Nevermind, I found the problem. It ought to be solved in the next pre-release.

[genodeftest]

Thanks!

@genodeftest which browser and version are you using?

That was Firefox 56.0 x86_64 from Fedora 26 repositories.

[shirishag75]

It took quite sometime to find this tracking bug as others are closed but still mentioned in bugs/tickets elsewhere. I did comment on one but not others.

[genodeftest]

The latest prerelease version does not bootstrap correctly, see #862.

[LiamZ]

Hello,

I'm glad to see that RP maintenance is still handled by someone. Actually I did not see this ticket before a few days ago, and had started on rewriting RP in WebExtension myself (with my little JS experience). I thought it would require a complete rewrite for the transition from XUL/XPCOM. But from reading this conversation, it seems not.

I don't even see how Firefox is going to trash the old model anytime soon, with so much extensions not being migrated yet.

[myrdd]

Hi @LiamZ

and had started on rewriting RP in WebExtension myself

did you publish some code? are you interested in helping me with migrating? i could need some help :)

I thought it would require a complete rewrite for the transition from XUL/XPCOM

not all code needs to be rewritten. of course, all the api stuff must be changed, but I'm doing it step-by-step.

I don't even see how Firefox is going to trash the old model anytime soon, with so much extensions not being migrated yet.

I think they will…

[LiamZ]

I didn't publish anything yet. I just have a WebExtension written from scratch that displays a panel similar to the official one. A request count is displayed too (I found that easier to setup than I thought). But all that is just a small part.

I would be glad to be of help. I'll have trouble understanding the existing XPCOM code, having never made an add-on myself.

I'll check the dev channel to see if I'm able to assist.

[myrdd]

I would be glad to be of help.

Great! :) Let's switch communications to #826.

[baptx]

If someone can't wait for the WebExtension, there is uMatrix that can prevent CSRF and protect privacy by blocking third-party requests.

[genodeftest]

And also uBlock Origin's expert mode. I don't think RequestPolicy is much different here so I'm not sure whether it will just be duplicate work.

[myrdd]

And also uBlock Origin's expert mode. I don't think RequestPolicy is much different here so I'm not sure whether it will just be duplicate work.

https://github.com/RequestPolicyContinued/requestpolicy/issues/692#issuecomment-160327720:

The main, inherent difference between RP and all other blockers I know is the "other origins" feature.

[shirishag75]

Sharing as one who has used RP and then RPC quite a bit. While I did use uorigin and some of the other extensions as well, I found the interface easier. I haven't tried out umatrix and probably would hang around to see if the WE port gets completed before trying something else.

What I found easy (after the initial learning curve) is that RPC gives a great service. One of the few quibbles I have is sometimes I have to give .domainname for origin and sometimes .domain.subdomain or something along those lines and it's a bit of hit and trial at times to see which will work to get rid of third-party requests to a site.

[jrrdev]

@shirishag75 I totally agree with you, RPC is a great tool but sometimes it is difficult to find which domain to allow to not screw up entirely a website (for instance CDN or any domains used to serve CSS or images). Maybe after the port one possible enhancement may be to indicate what kind of resources is queried from a given domain. The WE API gives these info in the details param of webRequest.onBeforeRequest (see the type attribute which is a webRequest.ResourceType). If you folks agree with that, the best would be to open a separate issue I guess :-)

[myrdd]

I plan to finish WE transition by the end of Feb'18 at the latest. If you are a developer, please consider helping with the transition. Thank you @jrrdev for your contributions so far! 👌 :)

[Make42]

I just wanted to let you know, that I am really looking forward to see RequestPolicy in Firefox Quantum! It's the best add-on I a have and now I can't use it. So sad :-(.

[yellowpattern]

When I click on the URL at the top of this page, I do not see a "developer channel" link.

If I go to https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/versions/beta then all I see are "This add-on is not compatible". Downloading a file and doing a manual install does not work.

Upgrading to 57 is the only way to combat the Spectre bug in Firefox.

[genodeftest]

@yellowpattern wrote:

Upgrading to 57 is the only way to combat the Spectre bug in Firefox.

If you are not using Firefox 57, you really should be using Firefox ESR 52.x, which is safer to use and not affected by at least one of the Spectre bugs.

[myrdd]

If I go to https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/versions/beta then all I see are "This add-on is not compatible".

What browser and version are you using? Navigate to about:support if you don't know.

[genodeftest]

@myrdd : I can reproduce the "This addon is not compatible" issue with Firefox 57, which is expected.

[shirishag75]

JFTR I made a blog post about RPC at https://flossexperiences.wordpress.com/2018/01/06/site-genuiness-and-requestpolicy-continued/ . While this probably is not the best place to mention it, I am tired hence putting it here. Feel free to comment if there might be something I missed.

[myrdd]

@genodeftest you're right, that's expected.

@shirishag75 if you want, you can create a new wiki page called "external resources" and put a link to your blog post there. :)

[myrdd]

I've created issue https://github.com/RequestPolicyContinued/requestpolicy/issues/884, which should be quite easy to work on with just a little JS knowledge.

[gitthisissue]

FYI - there are 2 addons which work on FF57. This and this.

Source: https://addons.mozilla.org/en-US/firefox/search/?q=RequestPolicy

[YtvwlD]

And uBlock Origin works quite similar, if you configure it that way (remove all lists, set "external requests" to "deny" for all sites).

[DuMuT6p]

uBlock Origin in advanced mode or uMatrix both have RequestPolicy functionality. However RequestPolicy had similar interface as NoScript so it was easier to understand. Hope the F57 version doesn't differ much.

@myrdd I can't find a pre-release version for F57 in AMO even trough it says in description. Is there one currently ? If not please let us know if/when we can help with beta tests. I really appreciate your time and effort. Thank you for continuing the addon.

[myrdd]

uBlock Origin in advanced mode or uMatrix both have RequestPolicy functionality.

Mostly. They do not have the concept of "other origins".

I can't find a pre-release version for F57 in AMO even trough it says in description. Is there one currently ?

The pre-release version is not yet Fx57+ compatible.

If not please let us know if/when we can help with beta tests.

I will. Thank you! :)

[YtvwlD]

They do not have the concept of "other origins".

At least uBlock Origin has.

There's a toggle for resources from third-party sites.

[myrdd]

Let's please discuss RP and other addons there: https://github.com/RequestPolicyContinued/requestpolicy/issues/692#issuecomment-357901591

[shirishag75]

I guess we just have to wait. it out. There doesn't seem to be any new commits after the last commit on 18th December 2017 https://github.com/RequestPolicyContinued/requestpolicy/commits/develop

[myrdd]

New pre-release available here: https://addons.mozilla.org/en-US/firefox/addon/requestpolicy-continued/versions/1.0.beta13.2.2106.r3a96ef0aa.pre

No new features, just a step towards the WE port.

Thanks to @jrrdev, who contributed to this pre-release.

[myrdd]

From now on, the pre-releases will be available here: https://sslsites.de/requestpolicy.256k.de/requestpolicy-nightly.xpi Be sure to disable or uninstall any other RP/RPC versions!

[myrdd]

A new pre-release is available. Please install/use it and report any issues.

(weApi.storage and weApi.privacy.network are done, see https://github.com/RequestPolicyContinued/requestpolicy/issues/893)

[ansell]

I get an error about compatibility when I attempt to install it:

[myrdd]

@ansell yes, you still need Firefox <= 56, sorry

[cheggerdev]

What is the plan about the first (pre-)release that is supposed to work with Firefox > 56 ?

[shirishag75]

I am just a user not a developer.

@myrdd what is the version of requestpolicy.xpi at https://sslsites.de/requestpolicy.256k.de/requestpolicy-nightly.xpi

I had to do the following -

$ wget https://sslsites.de/requestpolicy.256k.de/requestpolicy-nightly.xpi
--2018-05-21 07:14:04--  https://sslsites.de/requestpolicy.256k.de/requestpolicy-nightly.xpi
Resolving sslsites.de (sslsites.de)... 80.67.16.21
Connecting to sslsites.de (sslsites.de)|80.67.16.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 394409 (385K) [application/x-xpinstall]
Saving to: ‘requestpolicy-nightly.xpi’

requestpolicy-nightly.xpi      100%[=================================================>] 385.17K   271KB/s    in 1.4s    

2018-05-21 07:14:07 (271 KB/s) - ‘requestpolicy-nightly.xpi’ saved [394409/394409]

$ mkdir requestpolicy
shirish@debian:~$ mv requestpolicy-nightly.xpi requestpolicy

and then -

$ cd requestpolicy/

~/requestpolicy$ unzip requestpolicy-nightly.xpi

I did see this in install.rdf -

<!-- Firefox -->
    <em:targetApplication>
      <Description>
        <em:id>{ec8030f7-c20a-464f-9b0e-13a3a9e97384}</em:id>
        <em:minVersion>45.0</em:minVersion>
        <em:maxVersion>56.*</em:maxVersion>
      </Description>
    </em:targetApplication>

I am trying to figure out the versioning of requestpolicy-nightly.xpi and not knowing where to look as well as a Changelog.

Update - I found the version info. as RequestPolicy Continued 1.0.beta13.2.2277.r4d80b1f76.pre but that was by installing it :(