search

gorhill / uBlock

uBlock Origin - An efficient blocker for Chromium and Firefox. Fast and lean.
GNU General Public License v3.0
47.57k stars 3.16k forks source link

A custom filter not working on uBO v1.13.11rc0 / v1.14.0, but working on 1.13.8 #2938

Closed gomoku closed 7 years ago

gomoku commented 7 years ago

Filter issues MUST NOT be reported here. Read first: https://github.com/gorhill/uBlock/blob/master/CONTRIBUTING.md

I don't think it's a typical filter issue, because the same filter is working on one version of uBO and not working on another one.

Describe the issue

A custom exception filter: @@||*.dcs.redcdn.pl^$domain=dcs.redcdn.pl|www.tvn24.pl not working (not bypassing) another (static regex) filter: /(\.log\.|stuk\.|\/st\.gif|\/up\.|js_error|omniuni|2o7|audit\.|log\.(go|js)|mint|(u|smart)target|counter\.|cgi-bin\/counter|sageanalyst|brt\.|[^n]dcs\.|\/counter|touchclarity|directresponse|danasoft|advance\.)/ from http://hostsfile.mine.nu/downloads/adblock.txt on website http://www.tvn24.pl/ on FF 55.0.3 + uBO v1.13.11rc0 and Opera 47 + uBO v1.14.0 But the custom exception filter working fine on FF 55.0.3 + uBlock Origin v1.13.8

One or more specific URLs where the issue occurs

http://www.tvn24.pl/ (more websites might be affected, I didn't checked)

Screenshot in which the issue can be seen

https://s3.dodajfote.pl/2017/08/30/screen108402.png https://s3.dodajfote.pl/2017/08/30/screen229a09.png

Steps for anyone to reproduce the issue

  1. On a brand fresh browser profile install the affected uBO version
  2. In uBO in "3rd party filters" tab untick all filter lists, except "My filters" (just to make sure, but the issue is also reproductible without unticking default filters lists)
  3. In uBO in "3rd party filters" tab add http://hostsfile.mine.nu/downloads/adblock.txt
  4. In uBO in "My filters" tab add the custom filter. (instead of 3 & 4 may alternatively load the uBO backup snapshot via uBO "Settings" tab)
  5. Apply the settings
  6. Visit the website.
  7. Notice the lack of images / the custom filter not working
  8. Repeat all the steps with uBO 1.13.8
  9. Notice the images showing / the filter working fine

Your settings

Your filter lists

http://hostsfile.mine.nu/downloads/adblock.txt

Your custom filters (if any)

@@||*.dcs.redcdn.pl^$domain=dcs.redcdn.pl|www.tvn24.pl (more filters might be affected, I didn't checked)

Additional snapshot of backup of uBlock Origin settings:

``` { "timeStamp": 1504122557251, "version": "1.13.11rc0", "userSettings": { "advancedUserEnabled": false, "alwaysDetachLogger": false, "autoUpdate": true, "cloudStorageEnabled": false, "collapseBlocked": true, "colorBlindFriendly": false, "contextMenuEnabled": true, "dynamicFilteringEnabled": false, "externalLists": "http://hostsfile.mine.nu/downloads/adblock.txt", "firewallPaneMinimized": true, "hyperlinkAuditingDisabled": true, "ignoreGenericCosmeticFilters": false, "largeMediaSize": 50, "parseAllABPHideFilters": true, "prefetchingDisabled": true, "requestLogMaxEntries": 1000, "showIconBadge": true, "tooltipsDisabled": false, "webrtcIPAddressHidden": false }, "selectedFilterLists": [ "user-filters", "http://hostsfile.mine.nu/downloads/adblock.txt" ], "hiddenSettingsString": "assetFetchTimeout 30\nautoUpdateAssetFetchPeriod 120\nautoUpdatePeriod 7\nignoreRedirectFilters false\nignoreScriptInjectFilters false\nmanualUpdateAssetFetchPeriod 2000\npopupFontSize unset\nsuspendTabsUntilReady false\nuserResourcesLocation unset", "netWhitelist": "about-scheme\nbehind-the-scene\nchrome-extension-scheme\nchrome-scheme\nmoz-extension-scheme\nopera-scheme\nvivaldi-scheme", "dynamicFilteringString": "behind-the-scene * 3p-frame noop\nbehind-the-scene * 3p noop", "urlFilteringString": "", "hostnameSwitchesString": "", "userFilters": "@@||*.dcs.redcdn.pl^$domain=dcs.redcdn.pl|www.tvn24.pl\n", "filterLists": { "assets/user/filters.txt": { "off": false }, "http://hostsfile.mine.nu/downloads/adblock.txt": { "off": false }, "assets/ublock/filters.txt": { "off": true }, "assets/ublock/privacy.txt": { "off": true }, "assets/ublock/unbreak.txt": { "off": true }, "assets/ublock/badware.txt": { "off": true }, "assets/ublock/experimental.txt": { "off": true }, "assets/thirdparties/easylist-downloads.adblockplus.org/easylist.txt": { "off": true }, "assets/thirdparties/easylist-downloads.adblockplus.org/easyprivacy.txt": { "off": true }, "assets/thirdparties/www.malwaredomainlist.com/hostslist/hosts.txt": { "off": true }, "assets/thirdparties/mirror1.malwaredomains.com/files/justdomains": { "off": true }, "assets/thirdparties/pgl.yoyo.org/as/serverlist": { "off": true } } } ```
gorhill commented 7 years ago

I can reproduce. Investigating.

gorhill commented 7 years ago

Fixed in 1.14.2. Thanks for taking the time to file a detailed report.

gomoku commented 7 years ago

Np, thanks too for the quick fixing.

I've had to fill also another similiar filter issue, but it was too late and I've gone sleep, now I've woke up and I've realised I don't need anymore to fill that another filter issue, because it was being caused by the current issue, thus it has been fixed along with the current issue, by 1.14.3b0 (didn't checked others 1.14.* yet). I only post here the affected filters & filters list in case it may help you in anyway:

The issue I've filled on the filter list bug issue tracker several days ago: https://github.com/azet12/PopupBlocker/issues/78

The cosmetic filter pl###sidebar-boxes from filter list: https://github.com/azet12/PopupBlocker/blob/master/PPB.txt was not working on https://www.darmoweogłoszenia.pl/, thus was not blocking the annoying popup banner at the right side of the webpage, as seen at: https://s3.dodajfote.pl/2017/08/31/screen36c981.png

Also similiar filters generated by me for debuging purposes were not working: .pl###sidebar-boxes www.darmoweogłoszenia.pl###sidebar-boxes www.xn--darmoweogoszenia-ryc.pl###sidebar-boxes darmoweogłoszenia.pl###sidebar-boxes xn--darmoweogoszenia-ryc.pl###sidebar-boxes darmoweogoszenia-ryc.pl###sidebar-boxes

Generally putting any domain before the filter was causing the filter not working, for ex. this one has been working: ###sidebar-boxes, the affected filters wern't working on: Firefox + uBlock Origin 1.13.8 but were working on: Opera + uBlock Origin 1.13.8 Firefox + AdBlockPlus 2.9.1 so it was only a Firefox's side affected issue.

gorhill commented 7 years ago

I woke up realizing the fix here is not the right fix. I wanted to keep the code change minimal, but the fix here could cause false positives. I will need to go with the less trivial code change.

gorhill commented 7 years ago

The cosmetic filter pl###sidebar-boxes from filter list: https://github.com/azet12/PopupBlocker/blob/master/PPB.txt was not working on https://www.darmoweogłoszenia.pl/

Works for me, both in Chromium and Firefox. Use the logger to diagnose such issue -- it's reported in the logger on my side. It's easy sometimes to forget that we may have switched off cosmetic filtering in the popup panel. In any case, completely unrelated to the issue here, these filters are handled in two separate filtering engine.

gomoku commented 7 years ago

Hey, I've been just right now switching between various versions of uBlock Origin to check some things here and there, and I've realised that something wrong is going on here with that filter issue:

Suddenly now on FF 53.0.3 + uBlock Origin 1.13.8 the affected filter: @@||*.dcs.redcdn.pl^$domain=dcs.redcdn.pl|www.tvn24.pl is working again, and also it has reminded for me now, that the filter indeed has been also working on v1.13.8 before, I just remeber that exactly. So the point is that the filter issue on v1.13.8 happens randomly - the filter sometimes works and sometimes not, which doesn't change the fact it had to be fixed.

As for the another filter issue: pl###sidebar-boxes, I've already been checking in the logger before, but I've checked the issue again right now, and the issue still persist:

FF 55.0.3 32bit + completely new fresh profile + uBlock Origin v1.13.8 freshly installed as the only extension + I've even used "Restore defaults button..." (but that's not required) + then unticked all filter lists except "My filters" where I've put a single filter: pl###sidebar-boxes + clicked "Apply settings" button + cleared the cache & cookies of https://www.darmoweogłoszenia.pl/ + visit https://www.darmoweogłoszenia.pl/

and the banner still popups, in the logger no yellow dom blocked element is showed.

It might be that this filter issue, like the first one, also happens randomly (at least for you), because for me it doesn't - the filter doesn't work all the time, wherein ###sidebar-boxes does.

gorhill commented 7 years ago

How many filters are reported being enforced in the "3rd-party filters" pane?

The "happens randomly" is just a convenient explanation when we don't know why something happens or does not happens. Code is deterministic, and knowing the code, there is no reason for me to believe there is an issue with cosmetic filtering, that would be a much more widespread issue, and so far this happens to you only. Ideally more people would check on their side and report, so far I do not have an issue on my side with your filter:

a

gorhill commented 7 years ago

Never mind, I can reproduce with 1.13.8/legacy, not with 1.14.2/webext.

gorhill commented 7 years ago

Ok the issue was already reported: #1937. It's one of those issue which I just ended up forgetting about. Despite the issue gone with the webext version, I still want to fix it for the sake of whoever still want to use legacy.

gorhill commented 7 years ago

Ok, issue is already solved, I just didn't realize it. You just need version 1.13.10+.

gomoku commented 7 years ago

Well, good to see, I also was thinking before in my mind that it could be something related to 'ł' in the domain's name. Hmm and looking at https://github.com/azet12/PopupBlocker/issues/78 at:

curl: (6) Could not resolve host: www.darmoweogłoszenia.pl
curl: (6) Could not resolve host: darmoweogłoszenia.pl

the same issue may happens to curl or I use bad version of curl.

Btw by "happens randomly" I mean the same what the guy mean: https://github.com/gorhill/uBlock/issues/2947#issue-254784568

@vahhadbe: And not 100% of the time either Doesn't happen 100% of the time

And what you mean: https://github.com/gorhill/uBlock/issues/150#issuecomment-97475945

@gorhill: occasional failure

From the code perspective they are deterministic, but by "happens randomly" I mean from the reality perspective, from the other unknown factors perspecitve (in this particular case it has turned out to be the latin char), because I was (and I do still) exepriencing in my life comical, ridiculous, completely random, unpredictable, hard to reproduce, nasty bugs in which I myself could not believe, how can they ever happen.

gomoku commented 7 years ago

On FF 55.0.3 32bit / Opera 47.0.2631.71 64bit (PGO) + uBO v.1.13.10 installed from https://addons.mozilla.org/pl/firefox/addon/ublock-origin/ uBO v1.14.2 webext installed from https://github.com/gorhill/uBlock/releases/download/1.14.2/uBlock0.webext.xpi uBo 1.14.3b0 installed from https://addons.mozilla.org/pl/firefox/addon/ublock-origin/versions/beta

I've notcied that network filters like:

! 31.08.2017, 17:37:18 xn--darmoweogoszenia-ryc.pl
||darmoweogłoszenia.pl/pl-mapa.js$script,domain=www.xn--darmoweogoszenia-ryc.pl

are not working on latin domains like https://www.darmoweogłoszenia.pl/

The script is not blocked, in the logger no red blocked script is showing up. The issue is not present on uBO v1.13.8 installed from https://addons.mozilla.org/pl/firefox/addon/ublock-origin/versions/

Static filters like pl###sidebar-boxes do work, but in the logger, when click on yellow blocked dom, an empty white field is showed instead of a name of the filter list the blocking script comes from ("My filters" in this case) : https://s3.dodajfote.pl/2017/08/31/screen513d1b.png

gomoku commented 7 years ago

(uBO v1.14.3b1 webext)

||darmoweogłoszenia.pl/pl-mapa.js$script,domain=www.xn--darmoweogoszenia-ryc.pl still not working, no red blocked script is showing up in the logger

pl###sidebar-boxes fixed, but www.darmoweogłoszenia.pl###sidebar-boxes still showing the empty blank white box when clicked in the logger

gorhill commented 7 years ago

Unable to reproduce your issues:

a

gorhill commented 7 years ago

Ok never mind, I tried with Nightly. I can reproduce with Firefox 55.

gorhill commented 7 years ago

This is a Firefox issue really: https://bugzilla.mozilla.org/show_bug.cgi?id=945240, fixed in Firefox 56.

gorhill commented 7 years ago

You mind opening a separate issue for the ||darmoweogłoszenia.pl/pl-mapa.js$script,domain=www.xn--darmoweogoszenia-ryc.pl case?

I want to fix this asap -- I consider this a serious regression, but I want to be able to refer to a separate issue, as it's completely unrelated to the original issue in comment 1. No need to fill it too formally, just fill it with https://github.com/gorhill/uBlock/issues/2938#issuecomment-326658126, except don't import the cosmetic filter issue, that is yet another one with no real severity.

gomoku commented 7 years ago

OK, done: https://github.com/gorhill/uBlock/issues/2950

gomoku commented 7 years ago

Just preventively to let you not forget, there is still last remaining issue to fix, the one with cosmetic filters containing domains containing Unicode chars, like darmoweogłoszenia.pl###sidebar-boxes showing the blank white box when clicked in the logger, on FF 55.0.3 32bit & 56 Nightly 64bit (57 haven't checked)