Subscriptions: General discussion

nodiscc commented 9 years ago

It would be good to define guidelines about what should/should not be included in official subscriptions; and whether we allow third party subscriptions. This was discussed at https://github.com/RequestPolicyContinued/requestpolicy/issues/339, https://github.com/RequestPolicyContinued/requestpolicy/issues/343, https://github.com/RequestPolicyContinued/requestpolicy/issues/510.

(Having lots of rules in the default official subscrptions may harm peformance, make lists harder to maintain, and not benefit many users.)

Anyway, general discussion about subscriptions lists. Offtopic-but-related is also welcome.

drzraf commented 9 years ago

I'm thinking about a toolbar icon submenu :

It seems that your subscription server have these policies set for this website", select one of them:
- noads (wh: 17)
- light (wh: 5)
- noporn (bl: 3)

User select one of the policy for this website (and can then customize it by adding/removing rules)

If subscriptions are to be bundled (or, better, downloaded after ext installation), I think the best way to match user needs are:

general top100 websites
top100 list for each country (choose one)

Here are some stats, allowedOriginsToDestinations.log (being generated from):

sed -nr '/extensions.requestpolicy.allowedOriginsToDestinations/s/.*"(.*)"\);/\1/p' ~/.mozilla/firefox/*/p
refs.js|tr ' ' "\n"|sed '/^$/d' > allowedOriginsToDestinations.log

$ grep -o '\..*|' allowedOriginsToDestinations.log|sort|uniq -c|sort -n

     10 .gouv.fr|
     14 .tv|
     18 .blogspot.com.ar|
     22 .co.uk|
     25 .de|
     41 .com.ar|
     80 .net|
    241 .org|
    250 .fr|
    584 .com|

Actually, .com are more subject to cluttering webpage with useless crap than .org. ccTLD specific rules coming 2nd.

About unique 3rd party websites, I've accumulated 700 unique hosts, 50% (338) of them being these 12:

     10 |bp.blogspot.com
     13 |blogblog.com
     15 |amazonaws.com
     15 |google.fr
     16 |ajax.googleapis.com
     18 |blogger.com
     18 |googleusercontent.com
     18 |wikimedia.org
     20 |cloudfront.net
     38 |gstatic.com
     60 |googleapis.com
     97 |google.com

For the "which website to include", I thus think that a: global Top100 + per-country top100 (user choose its own), seems like an acceptable trade-off for default policies sets.

But this does not tell anything about which kind of policy the user wants:

w3m web-style policy
privacy oriented policy
...

I'm greatly in favor of allowing "personal" and multiples subscriptions, including:

http://localhost:8888
file:///tmp/mysubscription.json
(and even a socket stream :) unless policies can be reloaded by sending a SIGUSR1 to firefox)

I think that policies downloaded from server should be easily customizable (but does anyone want to enter the hell of merging updated upstream policies with user local changes ? if not, then changes are stored in another json file which take precedence over subscription)

nodiscc commented 9 years ago

I'm thinking about a toolbar icon submenu :

Let's not over-clutter the menu. Managing subscriptions should be done using the "Subscriptions" preferences page. The menu UI for subscriptions was discussed in https://github.com/RequestPolicyContinued/requestpolicy/issues/509#issuecomment-61985401

User select one of the policy for this website

Subscriptions should apply everywhere, not on a particular site. Users should be given the choice to override a particular subscription rule from the menu though.

If subscriptions are to be bundled (or, better, downloaded after ext installation), I think the best way to match user needs are: general top100 websites, top100 list for each country (choose one)

In the first time I think we should concentrate on subscriptions that allow requests (i.e. for users that use the Deny default policy) for websites to work properly. Focusing on top100 is a good idea IMHO.

I'm greatly in favor of allowing "personal" and multiples subscriptions

So it may be a good idea to add a "Add custom subscription from: field/button to the subscriptions prefs page, allowing the use to add an unofficial subscription from a URL (like AdBlock does) or a local file.

We could reference popular unofficial subscriptions lists in the README of this repo

drzraf commented 9 years ago

On Thu, Nov 13, 2014 at 08:12:59AM -0800, nodiscc wrote:

User select one of the policy for this website

Subscriptions should apply everywhere, not on a particular site. Users should be given the choice to override a particular subscription rule from the menu though.

I didn't got it, I was suggesting a way to easily load subscription policies for 1 website instead of relying of a (big) subscription file. After a couple of week of browsing using RP, a user should start to have a fairly good set of policies matching its use while avoiding being overwhelmed with tons of default rules.

One could imagine a checkbox "RP autodownload mode" a user would activate a couple of weeks which make RP automatically fetch existing policies for the websites the user visits during this "test period".

In the first time I think we should concentrate on subscriptions that allow requests (i.e. for users that use the Deny default policy) for websites to work properly. Focusing on top100 is a good idea IMHO.

That's what I thought too, sorry if I didn't made it clear enough. (I too use a "default: deny" policy)

nodiscc commented 9 years ago

Copying from https://github.com/RequestPolicyContinued/subscriptions/pull/6#issuecomment-62394918

we should define very clearly what to put into a subscription and what not to include before we do big changes

RequestPolicyContinued/requestpolicy#339 (comment) not everyone accept the same restriction from policy, for example, an external script may trigger a slideshow of a news website what is acceptable to be blocked for one, but not for another, I think there is place for distinct "policies" here.

@myrdd @reqsub I think changes should be kept to a minimum to just enable the pages to render properly. For example I am ok to add {"o":{"h":".ebay.de"},"d":{"h":".ebaystatic.com"}} as it affects the page rendering, but not {"o":{"h":".chromium.org"},"d":{"h":".google.com"}} as chromium.org works fine without requests to google.com.

myrdd commented 9 years ago

this comment is related to my previous post:

we should define very clearly what to put into a subscription and what not to include before we do big changes

Currently in chrome://requestpolicy/content/settings/subscriptions.html you can see the following subscriptions:

Privacy

deny_trackers | Block destinations that can track you as you browse the web

Usability

allow_sameorg | Allow destinations that belong to the same organization as the origin webpage allow_functionality | Allow requests that are needed for websites to work properly allow_embedded | Allow requests for embedded content such as images and videos

Browser

allow_mozilla | Allow requests related to Mozilla and Firefox websites allow_extensions | Allow requests that are needed for other extensions to work properly

_But_ allow_functionality, allow_mozilla and allow_extensions are currently not existing in this repo. I suggest adding allow_functionality.

How to distinguish between `allow_functionality` and `allow_sameorg`?

That should be easy. Are the servers from domain A and domain B from the same organization?

if yes it's a rule for allow_sameorg.
- [edit] note: if the request from domain A to B is not needed for functionality, it should not be allowed
if not it might be considered for allow_functionality

_But_ I think allow_functionality should not allow requests that simply grab a font from some other domain – this is rendering-related and there should be a subscription dedicated to "nice" rendering.

Another hint about "functionality": IMO we should add only essential functionality to that list. Elements like social bookmark button (such as twitter, facebook) could be seen as a bug or a feature. Therefore it shouldn't be in the functionality subscription.

(it would be good to summarize this discussion in the end)

nodiscc commented 9 years ago

I suggest adding allow_functionality. allow_functionality should not allow requests that simply grab a font from some other domain we should add only essential functionality to that list

Tracking this at https://github.com/RequestPolicyContinued/subscriptions/issues/9

myrdd commented 9 years ago

@nodiscc I've found and fixed a syntax error: https://github.com/RequestPolicyContinued/subscriptions/commit/468073bee74dceb95da9950ed2a5f0f02454df9c. Please check JSON syntax before committing. RP won't update the subscription if there's an error, e.g. this time the error was:

[RequestPolicy] [INFO] [INTERNAL] Failed to update subscription [Subscription official allow_functionality https://raw.githubusercontent.com/RequestPolicyContinued/subscriptions/master/official-allow_functionality.json]: SyntaxError: JSON.parse: unexpected character at line 176 column 2 of the JSON data

marzametal commented 9 years ago

How to export rules created by the user?... or even export settings altogether, similar to how the previous RequestPolicy extension handled it?

nodiscc commented 9 years ago

How to export rules created by the user

They are stored in ~/.mozilla/firefox/*.default/requestpolicy/policies/user.json.

myrdd commented 9 years ago

Hi @marzametal

or even export settings altogether, similar to how the previous RequestPolicy extension handled it?

Afaik RP 0.5.x only exported the rules (origin rules, destination rules and origin-to-destination rules). If you would like to be able to export/import settings, please create a new issue in the requestpolicy repo.

By the way, questions and discussions about RP itself should be done not here but at https://github.com/RequestPolicyContinued/requestpolicy/issues/484.

RequestPolicyContinued / subscriptions