Forcible /g00 adware insertion on newspaper websites

uBlock-user commented 7 years ago

URL(s) where the issue occurs

orlandosentinel.com sandiegouniontribune.com sun-sentinel.com mcall.com boston.com

Those are the ones I have seen so far, there may be more.

Describe the issue

Forcibly inserts g00 adware content and abuses window.location API if blocked by a filter like /g00^$important until it turns into a bad request.

Screenshot(s)

https://i.gyazo.com/86ab54811f6aaa1785b3d308566d6af6.png

Versions

Browser/version: [here] Chromium 57
uBlock Origin version: [here] 1.10.0

Settings

Default

Notes

1) This didn't happen when I visited the website few days ago, however it seems the website is infested with adware as of today, as it keeps trying to load the /g00 stuff when it fails the first time and it also inserts shitty adware cookies too.

2) Blocking inline script does stop the onslaught attack of /g00 however breaks pictures from loading and possibly other things.

gorhill commented 7 years ago

The /g00 stuff is Instart Logic's crap. I have no problem with perceiving this as crapware -- their code goes out of its way to work against end users, doing its best to try to turn user agents (browsers) into proprietary devices.

uBlock-user commented 7 years ago

Well is there any solution for this, besides from blocking the inline scripts ? Blocking inline scripts would be the last thing I wanna do, that's why I posted here.

gorhill commented 7 years ago

is there any solution for this

I will be able to answer when I have the time to investigate.

IsraeliAdblocker commented 7 years ago

I've investigated Instart Logic's crap for the past 3 hours, I now know how they work, how they communicate, how they implement on new customers and etc. I have a lot of information to reveal and I know the best non "cat & mouse" solution to fight them that we can implement right now.

@gorhill , If you can arrange private channel maybe an invisible thread on issues.adblockplus.org (just give me access I am using the same username there) I will post there all the details.

I don't want Instart Logic people to see my research report.

uBlock-user commented 7 years ago

@IsraeliAdblocker Please do. If these guys find success, soon all other major blogs and websites will be infested with it and we will be forced to block inline scripts every now and then, so far folks at easylist forum came up with a filter which no longer works and only worsens the situation.

okiehsch commented 7 years ago

example.com##script:inject(abort-on-property-write.js, I10C) works on my end. Example: Go to http://www.sandiegouniontribune.com You will get peppered with g00 requests. Now addsandiegouniontribune.com##script:inject(abort-on-property-write.js, I10C) no more g00 requests, at least on my end. Should work with all the mentioned domains.

gorhill commented 7 years ago

example.com##script:inject(abort-on-property-write.js, I10C) works on my end

Tried first site in list, orlandosentinel.com, and the site is rather broken, images won't display.

I suggest:

orlandosentinel.com##script:inject(wowhead.com.js)

Will await feedback.

okiehsch commented 7 years ago

orlandosentinel.com displays fine on my end, nothing appears broken, anyway, if I go to orlandosentinel.com, then clear cookies, add orlandosentinel.com##script:inject(wowhead.com.js), then reload this is the logger output filtered for "g00" g002

and this after I add orlandosentinel.com##script:inject(abort-on-property-write.js, I10C) g003 and like I said all the pictures display just fine on my end.

okiehsch commented 7 years ago

The only mentioned site that doesn´t work on my end is boston.com, but the issue there seems to be the filter boston.com##script:inject(i10c-defuser.js) in uBlock filters list, if I disable it, it also works on my end.

uBlock-user commented 7 years ago

sandiegouniontribune.com##script:inject(abort-on-property-write.js, I10C)

That does stop the onslaught attack /g00 attack, however manipulates and adds /g00/refferr/i to the domain at the address bar, and still adds referrer tracking cookies.

sandiegouniontribune.com##script:inject(wowhead.com.js)

This one stops the attack from happening at the root page, however cookies are still created and inserted to the browser and occasional /g00 ads get loaded silently after few mins.

okiehsch commented 7 years ago

Did you clear the cookies before you added the filter? Because I don`t see any. g004

Neither is there anything added to the domain. g005

uBlock-user commented 7 years ago

Yes I do, I have page opened in another tab. Please let me finish what I'm testing. Also by cookies I meant third-party cookies which are inserted as first party.

https://i.gyazo.com/d40c182c13f113fb41ddee2a4ac4d5fd.png

using Wowhead reduced the amount crap cookies being inserted however some are still inserted apart from the main domain, even when I'm blocking 3rd party cookies and site data.

Apparently wowhead isn't effective as I thought. I deleted all cookies/site data related to the site sandiegounion tribune and with wowhead filter reloaded again.

Website (after few secs) - https://i.gyazo.com/b5fd844104770562743a921908b52b26.jpg

Cookies - https://i.gyazo.com/513e196f14fa3587ec624ade2a5c3bcf.png

uBlock-user commented 7 years ago

Tested with sandiegouniontribune.com##script:inject(i10c-defuser.js)

same result as wowhead, ads manage to load after few secs and crap cookies are being inserted.

IsraeliAdblocker commented 7 years ago

@gorhill , Please tell me how can I privately share my research with you?

okiehsch commented 7 years ago

@gorhill I can, not reliably, sometimes it works, reproduce a broken orlandosentinel.com with the filter orlandosentinel.com##script:inject(abort-on-property-write.js, I10C).

So my previous post was inaccurate. The reason that I couldn´t reproduce was that I didn´t realize that I used Chrome 49 on that computer. I still can never reproduce a broken orlandosentinel.com with Chrome 49, but I can with Chrome 55. Sorry for any confusion I caused.

gorhill commented 7 years ago

orlandosentinel.com##script:inject(wowhead.com.js) works fine on my side: the first load, the page will redirect to a non-g00 version eventually, as it thinks the console is opened, and as a result the Instart Logic code stops doing crappy things (not unlike cockroaches running for hiding spots when turning on the light):

There are instances of URL with g00 in it, but it's just the URL of the document itself.

okiehsch commented 7 years ago

I now tested Edit: orlandosentinel.com##script:inject(abort-on-property-write.js, I10C and sandiegouniontribune.com##script:inject(abort-on-property-write.js, I10C) with Chrome 49, Chrome 55, Firefox 50.1.0 and Microsoft Edge. The problem you describe only occurs on Chrome 55 for me. It works fine on all other browsers. My OS is Windows 10.

gorhill commented 7 years ago

@IsraeliAdblocker See https://github.com/gorhill/uBlock/issues/1930#issuecomment-268005424.

uBlock-user commented 7 years ago

Tried this boston.com,mcall.com,sun-sentinel.com,sandiegouniontribune.com,orlandosentinel.com##script:inject(wowhead.com.js)

With that filter, I tested, both orlando and sandiego, the first load is very slow and takes a lot of time for the loading spinner to stop; still creates some g00 cookies. After refreshing the site upto 3 or 4 times, it becomes normal. /g00 redirection is still there, however it's like a popup defuser, it comes when you click and the URL resets back immediately like it never happened, atleast the website is browsable now. I have yet to test the remaining aforementioned ones for similar behaviour.

gorhill commented 7 years ago

The boston.com one still exhibits the issue with the wowhead.com.js scriplet (I will rename more appropriately eventually). I am investigating -- I added a scriptlet which defuses Instart Logic's ability to detect that the console is opened, so I can freely investigate using dev tools now.

uBlock-user commented 7 years ago

It seems spoofing user agent string to Firefox's ones works perfectly on Chromium. I'm using uMX for spoofing Firefox's UA and it does the job too. So only Chromium based browsers are affected by this.

Edit - sandiego one still loads slow and injects g00 cookies and other crapware cookies.

Edit2 - doesn't seem to work on sun-sentinel.com, loads ads even after spoofing the UA.

gorhill commented 7 years ago

So only Chromium based browsers are affected by this.

Yes: https://np.reddit.com/r/wow/comments/5exq2d/wowheadcom_sucking_bandwidth/dagbmie/. The server will serve a different document if Firefox (or "not Chrome").

gorhill commented 7 years ago

Essentially, the g00 URLs are obfuscated URLs to 3rd-parties that would normally be blocked by blockers:

uBlock-user commented 7 years ago

And those 3rd party urls leave their crap cookies with the help of the script which inserts the cookies as first party ? I already have the Block 3rd party cookies and site data activated, so that's the only way around to insert 3rd party data onto my browser.

gorhill commented 7 years ago

And those 3rd party urls leave their crap cookies with the help of the script which inserts the cookies as first party ?

Looks like this.

gorhill commented 7 years ago

I find ||boston.com^$inline-script seems to work fine.

okiehsch commented 7 years ago

There is no obvious site breakage but www.boston.com/video will not work, if you disable inline-script. There is no "g00" crap in the sourcecode of www.boston.com/video, so if you add @@||www.boston.com/video$inline-script the video site works.

uBlock-user commented 7 years ago

Except for boston, the rest of the lot breaks at root page with thumbnails for the articles and videos at any individual article.

ghost commented 7 years ago

timeanddate.com loads g00 too. Noticed when a video ad appeared. ##script:inject(abort-on-property-write.js, I10C) breaks the date selection menus that pop up for example on https://www.timeanddate.com/date/dateadd.html when clicking into a field. ||g00.timeanddate.com^$subdocument seems to work.

uBlock-user commented 7 years ago

Another one - chicagotribune.com

Blocking the /g00 profiler did make it browsable, still inserts the same crap as others.

gorhill commented 7 years ago

The Instart Logic's code contains a list of sites using their obfuscation scheme, all those listed here are in there -- including chicagotribune.com.

gorhill commented 7 years ago

I have a solution, now I have to decide how to make it available. I am thinking of maybe turning uBO-WebSocket into uBO-Extra which would contains all the code which goes beyond filter-based solutions to address some nastiness out there, including the one reported here.

uBlock-user commented 7 years ago

I'm not using uBO-WebSocket ext. By turning Websocket into Extra, will users have to install that extension ?

gorhill commented 7 years ago

Yes.

IsraeliAdblocker commented 7 years ago

Dear Raymond, (@gorhill) I understand your status, but I feeling uncomfortable to reveal my research info to Public eyes. Please send me an email to: israeliAdblocker@gmail.com, and I will send you what I have, I want you to have that info, you can decide to use it or not at your own choice.

Thanks.

gorhill commented 7 years ago

I feeling uncomfortable to reveal my research info to Public eyes.

Given what Instart Logic's technology does, I think there is a lot of value to make public all your findings. Their technology is extremely hostile to users, as it's also a way to bypass a user's wish to block third-party cookies, or even a user's wish to block undesirable servers using a hosts file. I can see broad public disapproval to the technology and we should not underestimate the shame factor. (The company behind the technology knows this, as the obfuscation stops as soon as an investigative user open the dev console).

gorhill commented 7 years ago

uBO-WebSocket has been renamed uBO-Extra, with a broader purpose of better meeting user expectations when they use uBlock Origin. It takes care of the issue here. Updated in Chrome store as well.

okiehsch commented 7 years ago

Works great with the mentioned sites, however with sites like http://ottawacitizen.com or http://www.thomson.co.uk it breaks part of the functionality of the sites. In the case of http://ottawacitizen.com you can't use the search function or sign in.

In the case of http://www.thomson.co.uk you can't use the interactive boxes. unbent Other sites with that issue montrealgazette.com, calgaryherald.com, edmontonjournal.com, theprovince.com, windsorstar.com, firstchoice.co.uk, leaderpost.com, thestarphoenix.com, falconholidays.ie

gorhill commented 7 years ago

Thanks @okiehsch, I will investigate the issues.

ottawacitizen.com: I did see the HtmlStreaming issue.
www.thomson.co.uk: looked fine (now it's "undergoing essential maintenance"). What was broken exactly?
montrealgazette.com: did not see the HtmlStreaming issue.
calgaryherald.com: did not see the HtmlStreaming issue.
edmontonjournal.com: did not see the HtmlStreaming issue.
theprovince.com: did not see the HtmlStreaming issue.
windsorstar.com: did not see the HtmlStreaming issue.
firstchoice.co.uk: Site is "undergoing essential maintenance".
leaderpost.com: did not see the HtmlStreaming issue.
thestarphoenix.com: did not see the HtmlStreaming issue.
falconholidays.ie: Site is "undergoing essential maintenance".

okiehsch commented 7 years ago

www.thomson.co.uk see second screenshot of my previous post. Edit: The Question of what was broken: You couldn't use the boxes "Fly from" "Where to" etc.
montrealgazette.com same issue as ottawacitizen.com, can still reproduce Same is true for the rest, except for the sites undergoing maintenance.

For example the console for leaderpost.com unbenannt1

gorhill commented 7 years ago

List of sites gathered from the IL's g00-related script (does not necessarily mean these sites are g00 infested, this will need confirmation):

about.com
applyabroad.org
boston.com
cargurus.com
chroniclelive.co.uk
cnet.com
corriere.it
gamepedia.com
mmo-champion.com
twincities.com
edmunds.com
foxnews.com
gamerevolution.com
holidaycheck.de
i10c.net
infinitiev.com
instarttest.com
drudgereport.com
headlinepolitics.com
refdesk.com
tellmenow.com
thepoliticalinsider.com
tmn.today
legacy.com
metal-hammer.de
msn.com
nasdaq.com
photobucket.com
calgaryherald.com
calgarysun.com
canoe.com
edmontonjournal.com
edmontonsun.com
financialpost.com
ifpress.com
leaderpost.com
montrealgazette.com
nationalpost.com
ottawacitizen.com
ottawasun.com
theprovince.com
thestarphoenix.com
torontosun.com
vancouversun.com
windsorstar.com
winnipegsun.com
ranker.com
reshadi.com
saveur.com
sherdog.com
slickdeals.net
space.com
buzznet.com
celebuzz.com
deathandtaxesmag.com
gofugyourself.com
idolator.com
spin.com
stereogum.com
thefrisky.com
thesuperficial.com
vibe.com
sporcle.com
sportingnews.com
testdomain.com
thinkfu.com
timeanddate.com
tronc.com
baltimoresun.com
capitalgazette.com
carrollcountytimes.com
chicagotribune.com
citypaper.com
courant.com
ctnow.com
dailypress.com
delmartimes.com
discoversd.com
growthspotter.com
hoylosangeles.com
lajollalight.com
latimes.com
mcall.com
orlandosentinel.com
ranchosantafereview.com
redeyechicago.com
sandiegouniontribune.com
southflorida.com
sun-sentinel.com
vagazette.com
trustedreviews.com
washingtonpost.com
weather.com
destinydb.com
hearthhead.com
lolking.net
mmoui.com
opshead.com
wowhead.com
zam.com
computershopper.com
extremetech.com
geek.com
ign.com
logicbuy.com
pcmag.com
speedtest.net

gorhill commented 7 years ago

montrealgazette.com same issue as ottawacitizen.com, can still reproduce

Looks like I might be served a different document, there is no instance of HtmlStreaming on my side in the source code, no such error at the console.

okiehsch commented 7 years ago

Seems to be the case unbenannt2

Edit: Your list of sites include quite a few where I can't see any remnants of g00 script in the source code.

gorhill commented 7 years ago

Ok I understand, they are browser-sniffing, and the g00 javascript is not served with Chrome 57, but occurs with Chromium 53 (I use Chrome to test uBO with default settings).

okiehsch commented 7 years ago

After a rather crude search here: https://publicwww.com/websites/i10c.morph/ All the sites listed there have at least remnants of "Instart Logic" in their source code.

Edit: thomson.co.uk, falconholidays.ie and firstchoice.co.uk have finished maintenance. All three use different source codes on my end and work fine now.

uBlock-user commented 7 years ago

uBO-Extra works as intended, thanks a lot. Still doubleclick.net and google-analytics's channel ID cookie was placed, can you do something about that ?

gorhill commented 7 years ago

I modified the approach re. g00: the g00-busting code will be injected only on sites for which it has been tested as working. Currently there is two version of the g00-busting scriptlet: one is the same as published yesterday, the other one is specific to those sites above using HtmlStreaming. The g00-busting scriplet will be injected only on site for which it is tested and confirmed as working as intended. Thus, report here any site which must be added.

ghost commented 7 years ago

uBO-Extra 2.0 works on timeanddate.com, uBO-Extra 2.1 does not. I assume it should be added to the list of working sites then. Is there an easier way to test sites than switching between 2.0 and 2.1?

gorhill commented 7 years ago

Is there an easier way to test sites than switching between 2.0 and 2.1?

Best is to load the extension locally, and add sites to the list in the code, then restart the extension to see if it works (clear cookies for the site -- I use a new private window for each test). There are two scriptlet versions for g00 stuff, uBO-Extra v2.1 contains both, while v2.0 contains only one. If view-source: for the site shows that it used HtmlStreaming, the second scriptlet is probably the one to use. Since timeanddate.com worked with v2.0, then the first version is the one to use.

In retrospect it was bad to apply the fix indiscriminately.

IsraeliAdblocker commented 7 years ago

@gorhill have you approached me via email? I've sent a response but now I have second thoughts that this weren't you, just checking.

uBlockOrigin / uAssets