solve URI scheme problems

[myrdd]

At the moment, RequestPolicy is designed to deal with Domain Names (wikipedia). But in fact RP has to deal with any kind of URI, and its syntax (see wikipedia) allows quite a lot.

One important part of the URI is the scheme; common examples are http, https and ftp, but it includes also about, mailto, view-source and many others. In fact, it could be any arbitrary string, like UT2004 for example (see #412).

As there are infinite possibilities for URI schemes, I'd like to generally block unknown schemes, but give the user the possibility to whitelist them. RP already has some hardcoded scheme rules (e.g. in shouldLoad) – the user will have the ability to extend them.

But, we have to keep one fact in mind: there are several non-http schemes which contain an http URI as its path (!). Example: view-source:https://github.com – view-source is the scheme, https://github.com is the path – this is how the specification is. In that case, we want to extract the "real URL" and check if this one is ok or should be rejected.

My proposal is to have several scheme lists:

• normal: URIs with such schemes are processed as always
• recursive: URIs with that scheme will be checked for "child" URIs
• whitelist: requests with destination URIs with such a scheme are allowed directly The user may add schemes to the whitelist only. Requests to unknown schemes will be blocked in general. (user will be informed)

Example schemes:

• normal: http, https, ftp
• recursive: view-source (already implemented (hardcoded))
• whitelist: {resource, about, data, chrome, moz-icon, moz-filedata, blob, wyciwyg, javascript} (see RequestPolicyService.prototype._isInternalRequest)

Known schemes: (list partly taken from here)

• about – https://github.com/RequestPolicyContinued/requestpolicy/issues/131, https://github.com/RequestPolicyContinued/requestpolicy/issues/377, https://github.com/RequestPolicyContinued/requestpolicy/issues/399, https://github.com/RequestPolicyContinued/requestpolicy/issues/426, (https://github.com/RequestPolicyContinued/requestpolicy/issues/427) – see also mozillazine and the about:about page
• chrome-extension (#571)
• feed – #395, #313, #386, #372
• gopher (#311)
• jar – https://github.com/RequestPolicyContinued/requestpolicy/issues/409
• magnet – #410
• mailto – #362
• mediasource – #512, RequestPolicy#448 – (or blob?) for Media Source Extensions
• mediastream
• nxm
• spotify (#296)
• UT2004 (#412)

[myrdd]

I still think this is the way to go, but I'd use different names for the scheme types:

• host schemes – they'll be processed by RP
• non-host schemes – they'll be always permitted
• recursive schemes – they'll be checked for embedded schemes (host or non-host)

If an unknown scheme is detected, the user will see a bar which shows the scheme and buttons to allow or temporarily allow that scheme.

[nodiscc]

I've re-read this and it looks good. Should mailto: , magnet: and feed: go in the whitelist by default?

[myrdd]

Should mailto: , magnet: and feed: go in the whitelist by default?

mailto yes magnet no (or is this a recursive URI?) feed is recursive, so no whitelisting

[myrdd]

https://github.com/RequestPolicyContinued/requestpolicy/issues/409 seems to be a partly solution for this. In the pull request, nsINestedURI is used.

[myrdd]

I've temporarily worked around the problem with commit 583cb45. It will be included in 1.0.beta8. Currently it whitelists the schemes in the first comment. If an unknown scheme is detected in an URI without a host, the user gets notified. So if I wouldn't have whitelisted mediasource, the following notification bar would appear:

The report this / more info button leads to this issue.

The workaround should be removed not later than for the stable 1.0 release.

[nodiscc]

1.0beta8 solves the magnet: and RSS link problems for me. Great job.

If an unknown scheme is detected, the user will see a bar which shows the scheme and buttons to allow or temporarily allow that scheme.

So what is the plan to get rid of the workaround? I'd be fine with

Requestpolicy has blocked a request to a magnet: URI from this page [Allow only this time] [Allow magnet: requests from this domain] [Keep blocking]

[myrdd]

So what is the plan to get rid of the workaround?

IMHO what you propose is still a workaround (a better one though). I'm planning to integrate custom schemes into the menu. This is, an origin and a destination can be a host, or a scheme without a host, for example.

[nodiscc]

Ok. The only thing that may go wrong (but I may have misunderstood) is that the user doesn't notice RP has blocked his click on the special scheme link. I.e. RP normally blocks requests that have not been explicitely been requested by the user (extra resources on a page).

If I understand well, what you suggest means that, when the user clicks on some magnet link (try from this page, magnet links in comments don't work), a new entry will be added to the RP menu (destination: magnet:), and the user will likely not notice it (I have clicked on this link and it doesn't work) unless he opens the menu and sees the magnet: destination. Am I right?

Edit: total misunderstanding on my part. See below.

[myrdd]

when the user clicks on some magnet link (try from this page, magnet links in comments don't work), a new entry will be added to the RP menu

no, sorry for being unclear. After this issue has been solved, link clicks will always be allowed. The issue about mailto and magnet links was the following: RP recognized that it was a link click, so it called the allow() function. That function saves the request in a list so that all requests can later be used for display in the menu. However, RP currently assumes that all URIs have a host, but that's wrong. RP currently can save only requests whith a host. Because the (origin or destination) URI doesn't have a host, the allow() function throws an error which is then catched by shouldLoad(). Whenever shouldLoad() catches an error, it will block the request, even if the policy decision ("allow") was already made.

[nodiscc]

Ok. Nothing to see here, I misunderstood. Thanks for taking the time to explain.

[myrdd]

Thanks for taking the time to explain.

You're welcome :)

[johnp]

Reporting 'safari-extension' scheme as seen on mydealz.de. I really don't know what to do with this, but by the name it looks like to be something we don't need in Firefox anyway.

[myrdd]

Thanks for reporting this @Yoyo117. Certainly I can't reproduce it, but I also think that it is irrelevant, so it won't be whitelisted.

[ldgbc]

Not sure if this a Firefox issue, Greasemonkey or RequestPolicy issue. Firefox 34.0 work without the popup error

Mozilla Nightly 37.0a1 (2014-12-11) RP 1.0b8.1 Greasemonkey 2.3 I'm getting this error 'greasemonkey-script' scheme

Using this script: AdsBypasser (https://greasyfork.org/en/scripts/4881-adsbypasser) Using this site: http://imagetwist.com/jh0klc48z1xh/1.jpg.html (I upload this image for test) Will display the error; (I'm guessing) this it might be due to this code found here: https://greasyfork.org/en/scripts/4881-adsbypasser/code

    function dispatchByString (rule, url_3) {
      var scheme = /\*|https?|file|ftp|chrome-extension/;
      var host = /\*|(\*\.)?([^\/*]+)/;

Another script that also have this 'greasemonkey-script' error is: https://github.com/Pashe/8chan-X/raw/2-0/8chan-x.user.js Can be tested here: https://8chan.co/tech/res/60929.html I think it got to do with this code:

// @grant       GM_

These value use localstorage I think. Using "Disable blocking" still give this error also.

[djarb]

RequestPolicy breaks the FloatNotes addon, by disallowing requests to floatnotes:blank without providing any way to whitelist them.

[myrdd]

@djarb don't you get a notification like in the following screenshot?

If not, are you using the current release? https://github.com/RequestPolicyContinued/requestpolicy/releases

[myrdd]

Wait, sorry, I forgot that if that notification comes up, the request has been blocked! So I'll create a hotfix commit (beta8.2) which will whitelist the following schemes:

• gopher (#311)
• spotify (#296)
• greasemonkey-script (see also #562)
• floatnotes (https://github.com/RequestPolicyContinued/requestpolicy/issues/447#issuecomment-70780082)

I think the following extensions don't need to be allowed:

• safari-extension
• chrome-extension

I realize that whitelisting via commits isn't nice… If this issue is not fixed in the near future, a solution might be to have a setting in about:config for whitelisting schemes. That setting can be removed when this issue is fixed.

[myrdd]

I've released 1.0.beta8.2 which adds the schemes I've mentioned in the previous post to the temporary scheme-whitelist.

@djarb can you please check whether the FloatNotes issue is fixed?

[djarb]

Yes, FloatNotes is functional again.

[myrdd]

@djarb Great! Thanks for checking.

[salj]

I need a whitelist for URIs that look like "bankid:///" + some crypto tokens, which is needed for handling electronic signatures in Sweden through an external application.

[myrdd]

I've updated the milestone to 1.0.beta10. Not sure whether that's possible, but this issue should be fixed in the near future.

[mikecardwell]

Was sent here after seeing the following error message:

This page contains a request with a 'mediastream' scheme which us unknown to RequestPolicy. Please report it.

This page is not publicly available, but is easy to replicate. It contains a minimal video tag:

<video autoplay></video>

And a tiny bit of javascript which requests access to the webcam and feeds video/audio into that video tag:

navigator.mozGetUserMedia({audio: true, video: true}, function(stream) {
    var video = document.querySelector('video');
    video.src = window.URL.createObjectURL(stream);
}, function(){});

This works if I disable Request Policy.

EDIT: An extra bit of info I just uncovered. If I do an inspect element on the video tag after this JS has been run, it has a src attribute which contains this:

mediastream:http://www.example.com/22f6daa2-5295-4c31-b828-411f8c6ffafe

The hostname matches the site which the html and js came from. The UUID was automatically generated by the browser I guess. (I replaced the real hostname with www.example.com for this example)

[At-Libitum]

I have the BetterTTV extension installed which alters the look/feel of twitch.tv Everytime I visit a twitch channel, the yellow bar appears with the message that the page contains a request with a 'chrome-extension' scheme which it claims to not know about.

[OrdinaryMagician]

nexusmods uses the "nxm" scheme for downloading with mod managers.

[chrcoluk]

I keep seeing this message on sites.

This page contains a request with a http scheme, which is unknown to requestpolicy, please report it.

An example url is

http://forum.kitz.co.uk/index.php/topic,15161.msg281837/topicseen.html#new

[myrdd]

@chrcoluk weird, on that page there is an image like this:

<img src="http://" alt="" class="bbc_img" />

So whatever the reason is, there seem to be URLs without a host. In this case you can simply ignore that message.

[chrcoluk]

well the notifications are annoying, cant they be turned off?

[myrdd]

@chrcoluk I understand that, but currently it's not possible to turn the notifications off, sorry. I'm working on it.

[alexlehm]

I have received a "this page uses a "https" scheme which is unknown to RP" notification, when looking at the source I think this is because one url is "https://" (probably a mistake by the html author). That could probably be ignored completely since the host isn't present

[myrdd]

I think this is because one url is "https://" (probably a mistake by the html author).

Exactly. It's the same as with "http" above. So yes, it can be ignored.

[alexlehm]

sorry, i missed the http entry above, what I meant was it could be ignored silently

[cm-mc]

rp is blocking multi web search (ff addon: https://addons.mozilla.org/firefox/addon/multi-web-search/) scheme (http://multiwebsearch/?s=*)

[OrdinaryMagician]

I think requestpolicy might also be responsible for breaking the MEGA add-on.

[cm-mc]

Not sure about Mega, I can't load mega.co.nz, but mega.nz works, and if I change mega.co.nz addresses to mega.nz dl works ok, too.

[alexlehm]

@OrdinaryMagician there are some problems with urls using resource: as url and also ones that use an empty url (not sure where that comes from), usually this can be determined with the request log.

[OrdinaryMagician]

Actually, false alarm. Looks like the MEGA add-on breaking is just a problem with Nightly on Windows (as usual), works fine when I switch to Linux. (Dunno why I was sometimes seeing pop-ups for a "mega" uri scheme)

[cm-mc]

Any solution in sight for this problem? And if it can't be easily fixed, can we at least have RP filtering disabled for non-supported schemes, as a stop-gap measure? I really need this gone.

[myrdd]

My current plan is to allow rules that specify the scheme only, i.e. without host. In fact it's already possible to add such rules, but they are ignored. When this is done, I'll create a new release.

The current state (commit 486ad39) is that the special requests we're talking about (requests with URIs without a host) are at least shown in the request log.

As soon as rules for URIs without host are working, those special requests have to be shown in the menu, not only in the request log.

[cm-mc]

The link above is dead (https://github.com/RequestPolicyContinued/requestpolicy/tree/dev-restartless-and-e10s).

[myrdd]

Which link are you referring to @cm-mc?

[cm-mc]

The link to the restartless build (I assume it had some sort of solution for this problem?), but I see now it has been removed.

[myrdd]

In this issue I can't find where the restartless build has been mentioned.

I assume it had some sort of solution for this problem?

No, as far as I remember nothing has changed on the dev-restartless-and-e10s branch regarding this issue. But: with commit 486ad39 I've started solving the problem, see my comment above – https://github.com/RequestPolicyContinued/requestpolicy/issues/447#issuecomment-84228189.

it has been removed.

Yes, I've removed the branch because I had merged it into dev-1.0 previously. The merge commit is a8dc42e, which has two parent commits. Before the dev-restartless-and-e10s branch has been removed, commit cb7367d had been the HEAD of the branch.

[cm-mc]

Nvm then, thought this pointed to some sort of fix: myrdd referenced this issue 3 days ago restartlessness and e10s #489

[Alexander--]

Here is a page, that reproduces the issue for me: http://habrahabr.ru/company/dataline/blog/253609/#habracut. Not sure, what exactly triggers it — would be nice, of you were writing the faulty URI to browser console.

[cm-mc]

@Alexander--: At least that page seems to load correctly; multiwebsearch just gives me a blank page...

[shirishag75]

Came across this http://www.off-grid.net/tag/self-sufficiency and was dumped here by requestpolicycontinued.

[xelaseer]

Just came across a 'ms-windows-store' scheme for Windows Metro desktop apps. See here http://apps.microsoft.com/windows/en-ca/app/chinese-english-dictionary/51d94dcb-ae64-4277-9b2b-ae0a809a314e.

[cm-mc]

Also http://www.startmenu8.com/pt/index.html - the odd thing: this happened while rp blocking was disabled.

[anonsubmitter]

This page contains an unknown https scheme: https://marvel.com/comics I don't really understand how to gather more info on it though.