search

toshiya44 / myAssets

Custom filters and other resources to use with uBlock Origin and uMatrix.
GNU General Public License v3.0
14 stars 3 forks source link

All about Gelbooru (Unblocked ads, Site functional, Counter other filterlists, etc.) #1

Open kowith337 opened 7 years ago

kowith337 commented 7 years ago

because they ruined site itself by put the fool php document that load as script, but I guess blocking script will be break Translation Note function. Test Link: http://gelbooru.com/index.php?page=post&s=view&id=3406493

I think it should be unblocked, but normal unblock rules also unblock fool php docs, suggest to use this instead. @@||gelbooru.com/script/application.js? because an actual functional script have version remark after ? sign (you will see number 46 after that in logger)

and use this rule ||gelbooru.com$script,subdocument to block other unwanted scripts and subdocs that they will be put or change filename in future. *like they put frontend_loader.js and backend_loader.php in the past to start call ads from exoclick by host script locally instead of directly call from third party. (see ryanbr/fanboy-adblock#30)

toshiya44 commented 7 years ago

Thanks for pointing this out. Looks like @@||gelbooru.com/script/application.js? already exists in Easylist but I'll keep it in my list too just in case. Added ||gelbooru.com$script,subdocument . Let's see how it goes.

kowith337 commented 7 years ago

I've unload other filter lists except hosts based list and your list for a test, an application.js script still blocked. But while that script was blocked, I can't see any request that call application.js.php, guess it's depend on main script file. screenshot 88

And when I've unload your list and load Easylist filter, it still allow both application.js and application.js.php, plus all scripts from ads.exoclick.com screenshot 91 I don't know how to trim this because the rule is @@||gelbooru.com/script/application.js is not @@||gelbooru.com/script/application.js? << have question mark behind

expected result image

toshiya44 commented 7 years ago

Can you test again please? I added the question mark in the filter. It works on my side. commit #f18df9d

kowith337 commented 7 years ago

Confirm work as expected. sketch 6 sketch 7

In the image view page, I don't see any application.js.php request.

toshiya44 commented 7 years ago

Yup, I noticed that the site doesn't request for application.js.php sometimes. Not sure if it's a bug on their side.

kowith337 commented 7 years ago 
<script type="text/javascript">
    ExoLoader.serve({"script_url":"http://gelbooru.com/script/application.js.php"});
</script>

Later after everything loaded, this line below was generated. (Observed in Chrome F12 inspector)

<script async="" type="text/javascript" src="http://gelbooru.com/script/application.js.php"></script>
<script>
    ExoLoader.serve({"script_url":"http://gelbooru.com/script/application.js.php"});
  (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
  (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
  m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
  })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
  ga('create', 'UA-2246042-1', 'auto', {'sampleRate': 100});
  ga('send', 'pageview');
</script>

They tried to serve ExoClick ads with same method along with Google Analytics inception, but it cannot run script in that line, that's why no request for application.js.php in image view, what a poor coding...

toshiya44 commented 7 years ago

How did you figure out filters like these?

! Trim bottom space that reserved for advertisement.
gelbooru.com##div[style*="width:1000px"]:has(script)

I don't see them in uBO element picker.

kowith337 commented 7 years ago

Found in Ctrl+U source view, where the Exoloader.addZone inline-script called after image lists.

Because ##script:contains() that aim to remove nuisance inline-script isn't support in uBlock0 + Chrome.

kowith337 commented 7 years ago

I'm tested new rule to block attempt to inject application.js.php via script async, but I think it's make me found something new. I've found JuicyAds network in post list page.

<script async src="//adserver.juicyads.com/js/jads.js"></script>

It's not new, they serve JuicyAds long time ago, before they switched to use ContentABC, CCBill and ExoClick, also increase amount of advertisement services to throw in face of users as well.

toshiya44 commented 7 years ago

Looks like juicyads is already being blocked by the hosts files parsed by uBO. Still, I'll add a ||juicyads.com$important rule just in case they change the subdomain later on.

8103413331466

kowith337 commented 7 years ago

##script blocking rules.

  • Aim to remove inline-script to prevent and reduce third party request that you can see at per-site switching panel.
  • I can't deal with <script> that have src right now. 
    gelbooru.com##script:contains(_gaq.push)
gelbooru.com##script:contains(ad_idzone)
gelbooru.com##script:contains(adnOpt)
gelbooru.com##script:contains(ExoLoader)
gelbooru.com##script:contains(GoogleAnalyticsObject)
gelbooru.com##script:contains(trw_domain)

Bonus

gelbooru.com##noscript
gelbooru.com##script[async]
toshiya44 commented 7 years ago

I'm unable to get the ##script rules to work for some reason. Don't see any changes and network requests to tracking servers keep being made. Is this what you meant by "It seems can only block DOM to hide them, but cannot prevent access or execute."? It's not showing up in the logger either. (for other sites as well). Is this a bug? I'm on Firefox Nightly 53

kowith337 commented 7 years ago

Inline-script rules (##script:contains) doesn't show in log, but can confirm those line was blocked because some third party request will not show in network switch pane, e.g. AdNium, Google Analytics.

Compare Before | After

but some inline-script will be hardly noticed is it blocked or not, because some of them doesn't use to alternatively call other network requests, but just use for other function that sometime doesn't need to call other network requests. I've apply ##script:contains(ExoLoader) rule to prevent call ExoLoader.addZone function for generate space to preparing deploy ads when successfully called backend script (application.js.php) via ExoLoader.serve, the result is it cannot generate those space and cannot deploy ads.

toshiya44 commented 7 years ago

Okay. I figured out why I wasn't seeing any change. It's because my computer's hosts file is already blocking some of the ad servers, so they aren't loaded in my computer in the first place. Sorry for the confusion.

I'll add the script filters for so that people who don't have hosts file on their computer can block them too.

kowith337 commented 7 years ago

Video ads are returnes and seems to be blocked by EasyList, but I think I cannot found video ads anymore, even in logger.

But it seems good to retain blocking rules because it will return again someday, like host advertiser scripts as first party, JuicyAds, Intermission break.

Currently I cannot see any additional suspicious injection (both 1P and 3P) and it seems they cannot bypass anything for now (in Fennec F-droid 51.0 + uBo 1.10.7rc2) except a return of intermission ads that will takeover and force to view ads at least 10 seconds then redirect back to a page where you're.

This fix by visit intermission.php earlier before browsing/viewing images and refresh intermission page again after specific interval (e.g. browsed 10 pages or images)

Moreover, you can cut down more 3p server hits in Firefox Android and other Gecko-based by using NoScript anywhere, then un-whitelist other server except first-party one. I will close this issue now until they update site to inject ads again or found suspicious things that cannot handle by ABP but uBo.

kowith337 commented 7 years ago

I cannot reproduce in Firefox but in Chromium is still look fine... I think because of some Inline-script block caused this break. @kowith337/Gelbolube#2

toshiya44 commented 7 years ago

I don't have this issue for some reason. I also tried using your list instead of mine and still couldn't reproduce. I'm on Firefox Nightly 54.0a1 (2017-02-12) (64-bit) I'll try other versions. image

kowith337 commented 7 years ago

It's just my side because I've installed AdBlock Protector script in Firefox that currently not support Greasemonkey, that's why it caused notes break. Sorry about that.

kowith337 commented 7 years ago

Now most functions (include translate notes) will be depend on jQuery, ~~also legitmate application.js is no longer have ? behind. (observed in uBO logger) is this help?: https://github.com/kowith337/GelboLube/commit/6db2968a09d4df2a4fedff1fc83811a069020f23~~

Update, I've re-checked application.js again and found only ExoLoader function in that script, I think they really move everything to jQuery, now you can safely to block that. https://github.com/kowith337/GelboLube/commit/92b65283d5ae6363a24328f65de08f93b1c8cfb5

toshiya44 commented 7 years ago

You have to add @@||ajax.googleapis.com/ajax/libs/jquery/$script,domain=gelbooru.com too. https://github.com/toshiya44/myAssets/commit/66581b230fa4769636e1dab3fcedc4923a49e7bd

kowith337 commented 7 years ago

In my side, I didn't block Google resources (except analytics), but set up to use advance settings to block 3rd party in gelbooru. For now, i've set ajax.googleapis.com to noop to allow jQuery assets.

kowith337 commented 7 years ago

They revert back to old system

Edit: Guess it's a cookie that create by CloudFlare to store identity and prevent DDoS/spam, but I didn't see any requests that made to.

toshiya44 commented 7 years ago

Use uMatrix for cookies?

Old filters are working again. I guess I'll comment out the newer filters for now.

kowith337 commented 7 years ago

Use with and without. (Also set browser to block 3p cookies) And I have found XHR that connect to v.php?... when viewing full image, but still don't know how __cfduid cookie can be created. image

kowith337 commented 7 years ago

Noticed they serve Gelbooru site as HTTPS now, and seems they use CloudFlare SSL, maybe it's clear to untouch that cookie.

toshiya44 commented 7 years ago

Seems like it. Probably because of that new bill.

kowith337 commented 7 years ago

Sorry for not report here a while ago...

  1. I think it's fine to use @@||gelbooru.com/ads.js?$script exception because this file content is just set the advertVar or abvertDar value to not have it blank, the fallback inline-script advert will be show if that value blank. image

  2. They seems change ads location to /thumbnails/0C/, but I've decided to use this rule to stop future fake thumbs that located at same location. (mean under /thumbnails/) Explained at this commit. kowith337/Gelbolube@3fbe7df99efcfa3eb31538fd3d34f0f2e6b495a6

  3. Found banner ads sometime injected at the top of image list or single image, first I've found it's load from //assets.gelbooru.com/r19/ but forgot to copy target URL, only know it's promote JabComix, an adult comic site. kowith337/GelboLube@e1a2dd50592405cc6151ea2ca4228ad7d7af9638 #diff-4f5ac6c63238e04b01e42ea957a728ceR47

toshiya44 commented 7 years ago

Added some filters. I think it'd be best if we make fewer exception filters for gelbooru, as they might end up getting abused. Gelbooru people really likes changing layout all the time. At least they finally added https support. lol

kowith337 commented 7 years ago

That's why I've start to block every script and subdocument first, then allow only some if needed. Currently only allow 2 scripts (checkmarked is mean blocked by filterlist.)

For the possible of prevent /thumbnails/ ads image abuse, I've used this solution. ||gelbooru.com/thumbnails/*.gif|$image,important ||gelbooru.com/thumbnails/*.jpg|$image,important <- Conflict with other appearance

because advert images that placed at that location is have URL like this (obtained from inline-script) GIF: <img src="//assets.gelbooru.com/thumbnails/0C/108499-17215113904251300_1_xacd.gif"> JPG: <img src="//assets.gelbooru.com/thumbnails/0C/322388-17316102417302154_1.jpg">

Then look at the actual thumbnails in list view... <img src="//assets.gelbooru.com/thumbnails/5a/9f/thumbnail_5a9fa4a5b025fc542e3176153371304d.jpg?3641118" ...>

Actual thumbnails always requested as JPG with thumbnail_ file name beginning and have number that reference to PostID assigned after question mark ? sign (when viewing in post list) while those advert images that placed at /thumbnails/ doesn't have anything assigned after, and exactly ended with specific file type.

But also have this wildcard exception, in case if it's accidentally blocked. @@||gelbooru.com/thumbnails/*/*/thumbnail_*.jpg$image

kowith337 commented 7 years ago

kowith337/GelboLube#3 New site update make all thumbnails also use redirect.php, this cause all thumbnails hidden due to ##a[href*="redirect.php"] hiding rule.

toshiya44 commented 7 years ago

I imported some of the filters and the site seems to be working fine now. But, I every single thumbnail has a link like, https://gelbooru.com/redirect.php?s={Very long string of numbers/ID}== To me it looks like a hash ID. Are they using it to track which images are being viewed by individual visitors? It looks pretty fishy to me. Sankaku and danbooru will do for the time being...

kowith337 commented 7 years ago

Not at all, they first use redirect.php at the ads link first, then also apply on all thumbnails, maybe to track and count how many visitors who clicked ads. But they won't apply if you click thumbnail from image pools, recent comments, somewhere else that not at the post lists. I'm agree that it seems like some kind of tracking, but now they made by themselves. Probably need to create UserScripts to replace them with original link, I have some guideline that maybe can help to create it!

kowith337 commented 7 years ago

Good News: They stopped apply redirect links at posts list, all thumbnail posts list hyperlink is now link to actual URL now... Neutral News: New responsive design applied, seems very compatible for mobile browser users, also. Bad News: Perhaps of above, some blocked rules may obsolete or won't work, especially element hiding.

On my work: Currently

Both of false rules was blacklisted by badfilter at my side.

toshiya44 commented 6 years ago

Any idea what https://gelbooru.com/script/license.gelbooru.js is about? The site works fine with it blocked.

kowith337 commented 6 years ago

I don't know about it, also... It just partial encoded script, but it seems does nothing related to ads or functionality...

kowith337 commented 6 years ago

Any idea about this? kowith337/GelboLube#5

Update 1: Probably I need to stop using ##script:inject(abort-on-property-write.js, ExoLoader) and any kind of blocking exoLoader function for allow lazyload function to work properly, because it seems was hard-coded to depend on that function, but still not sure the exo_zones cookies will still deployed and cannot doing anything more than that?

Update 2: Allowing any kinds of ExoLoader is also allow popunder ads to be loaded as usual, seems like the great victory for them, though. (judge by the new announcement text that wrote on the red background)

Update 3: BetterJSPop is a new problem that caused to made popunder loaded (same method that ExoClick use, but currently just test to load TheDoujin site, are they made those scripts by themselves?) Now confirmed that you're fine to block other scripts that not named as jquery at the beginning, such as miscJs.js and license.gelbooru.js because it seems like a script that can re-execute inline script to make sure it can bypass uBlock scriptlets function.

Update 4: This should be whitelisted, too. ||gelbooru.com/script/lazyload.js?$script,first-party

Update 5: Please undone blocking miscJs because it used for autocomplete (reproduced) then add this exception... @@||gelbooru.com/script/miscJs.js?$script,first-party

toshiya44 commented 6 years ago

didn't notice that miscJs was responsible for autocomplete.