Evaluate adding the the Windows Spy Blocking hosts

[StevenBlack]

See https://github.com/crazy-max/WindowsSpyBlocker.

[FadeMind]

Maybe I will just merge https://github.com/crazy-max/WindowsSpyBlocker/tree/master/data/hosts/win10 content in hosts.extras/Telemetry?

[StevenBlack]

That would seem sensible, yes!

[FadeMind]

I will skip extra rules content

0.0.0.0 apps.skype.com
0.0.0.0 cdn.content.prod.cms.msn.com
0.0.0.0 choice.microsoft.com.nstac.net
0.0.0.0 client.wns.windows.com
0.0.0.0 client-s.gateway.messenger.live.com
0.0.0.0 clientconfig.passport.net
0.0.0.0 deploy.static.akamaitechnologies.com
0.0.0.0 device.auth.xboxlive.com
0.0.0.0 dmd.metaservices.microsoft.com
0.0.0.0 dns.msftncsi.com
0.0.0.0 g.live.com
0.0.0.0 img-s-msn-com.akamaized.net
0.0.0.0 licensing.mp.microsoft.com
0.0.0.0 login.live.com
0.0.0.0 msftncsi.com
0.0.0.0 oneclient.sfx.ms
0.0.0.0 pricelist.skype.com
0.0.0.0 s.gateway.messenger.live.com
0.0.0.0 s0.2mdn.net
0.0.0.0 sO.2mdn.net
0.0.0.0 search.msn.com
0.0.0.0 settings-ssl.xboxlive.com
0.0.0.0 static.2mdn.net
0.0.0.0 store-images.s-microsoft.com
0.0.0.0 storeedgefd.dsx.mp.microsoft.com
0.0.0.0 tile-service.weather.microsoft.com
0.0.0.0 time.windows.com
0.0.0.0 tk2.plt.msn.com
0.0.0.0 ui.skype.com
0.0.0.0 www.msftncsi.com

against issues with skype and other services

@StevenBlack

[StevenBlack]

Skype is a weird one.

Did you know... when I open a local Excel or Word file on Office 365 for Mac, these programs reach-out to connect to about twenty different domains, including Skype. I know this because I use little snitch.

I do not trust Microsoft; opening a local file should not trigger a 20-domain world tour.

[FadeMind]

@StevenBlack Here it is:

0.0.0.0 a-0001.a-msedge.net
0.0.0.0 a-0002.a-msedge.net
0.0.0.0 a-0003.a-msedge.net
0.0.0.0 a-0004.a-msedge.net
0.0.0.0 a-0005.a-msedge.net
0.0.0.0 a-0006.a-msedge.net
0.0.0.0 a-0007.a-msedge.net
0.0.0.0 a-0008.a-msedge.net
0.0.0.0 a-0009.a-msedge.net
0.0.0.0 a-0010.a-msedge.net
0.0.0.0 a-0011.a-msedge.net
0.0.0.0 a-0012.a-msedge.net
0.0.0.0 a-msedge.net
0.0.0.0 a.ads1.msn.com
0.0.0.0 a.ads2.msads.net
0.0.0.0 a.ads2.msn.com
0.0.0.0 a.rad.msn.com
0.0.0.0 ac3.msn.com
0.0.0.0 activity.windows.com
0.0.0.0 ad.doubleclick.net
0.0.0.0 adnexus.net
0.0.0.0 adnxs.com
0.0.0.0 ads.msn.com
0.0.0.0 ads1.msads.net
0.0.0.0 ads1.msn.com
0.0.0.0 aidps.atdmt.com
0.0.0.0 aka-cdn-ns.adtech.de
0.0.0.0 answers.microsoft.com
0.0.0.0 apps.skype.com
0.0.0.0 az361816.vo.msecnd.net
0.0.0.0 az512334.vo.msecnd.net
0.0.0.0 b.ads1.msn.com
0.0.0.0 b.ads2.msads.net
0.0.0.0 b.rad.msn.com
0.0.0.0 bingads.microsoft.com
0.0.0.0 bl3301-a.1drv.com
0.0.0.0 bl3301-g.1drv.com
0.0.0.0 bn1304-e.1drv.com
0.0.0.0 bn1306-e.1drv.com
0.0.0.0 bn1306-g.1drv.com
0.0.0.0 bn2b-cor001.api.p001.1drv.com
0.0.0.0 bs.serving-sys.com
0.0.0.0 by3301-a.1drv.com
0.0.0.0 by3301-c.1drv.com
0.0.0.0 c.atdmt.com
0.0.0.0 c.msn.com
0.0.0.0 ca.telemetry.microsoft.com
0.0.0.0 cache.datamart.windows.com
0.0.0.0 cdn.atdmt.com
0.0.0.0 cds26.ams9.msecn.net
0.0.0.0 ch3301-c.1drv.com
0.0.0.0 ch3301-e.1drv.com
0.0.0.0 ch3301-g.1drv.com
0.0.0.0 ch3302-e.1drv.com
0.0.0.0 choice.microsoft.com
0.0.0.0 choice.microsoft.com.nsatc.net
0.0.0.0 choice.microsoft.com.nstac.net
0.0.0.0 compatexchange.cloudapp.net
0.0.0.0 compatexchange1.trafficmanager.net
0.0.0.0 corp.sts.microsoft.com
0.0.0.0 corpext.msitadfs.glbdns2.microsoft.com
0.0.0.0 cp101-prod.do.dsp.mp.microsoft.com
0.0.0.0 cs1.wpc.v0cdn.net
0.0.0.0 db3aqu.atdmt.com
0.0.0.0 db3wns2011111.wns.windows.com
0.0.0.0 db5.wns.windows.com
0.0.0.0 db5sch101100917.wns.windows.com
0.0.0.0 db5sch101101511.wns.windows.com
0.0.0.0 db5sch101101939.wns.windows.com
0.0.0.0 db5sch101110626.wns.windows.com
0.0.0.0 db5sch101110634.wns.windows.com
0.0.0.0 db5sch103082111.wns.windows.com
0.0.0.0 db5sch103082406.wns.windows.com
0.0.0.0 db5sch103092209.wns.windows.com
0.0.0.0 db5wns1d.wns.windows.com
0.0.0.0 db6sch102091105.wns.windows.com
0.0.0.0 db6sch102091412.wns.windows.com
0.0.0.0 dev.virtualearth.net
0.0.0.0 df.telemetry.microsoft.com
0.0.0.0 diagnostics.support.microsoft.com
0.0.0.0 ec.atdmt.com
0.0.0.0 ecn.dev.virtualearth.net
0.0.0.0 eu.vortex.data.microsoft.com
0.0.0.0 fe2.update.microsoft.com.akadns.net
0.0.0.0 fe3.delivery.dsp.mp.microsoft.com.nsatc.net
0.0.0.0 feedback.microsoft-hohm.com
0.0.0.0 feedback.search.microsoft.com
0.0.0.0 feedback.windows.com
0.0.0.0 flex.msn.com
0.0.0.0 g.msn.com
0.0.0.0 geo-prod.do.dsp.mp.microsoft.com
0.0.0.0 geover-prod.do.dsp.mp.microsoft.com
0.0.0.0 h1.msn.com
0.0.0.0 h2.msn.com
0.0.0.0 i1.services.social.microsoft.com
0.0.0.0 i1.services.social.microsoft.com.nsatc.net
0.0.0.0 inference.location.live.net
0.0.0.0 kv101-prod.do.dsp.mp.microsoft.com
0.0.0.0 lb1.www.ms.akadns.net
0.0.0.0 live.rads.msn.com
0.0.0.0 ls2web.redmond.corp.microsoft.com
0.0.0.0 m.adnxs.com
0.0.0.0 m.hotmail.com
0.0.0.0 mobile.pipe.aria.microsoft.com
0.0.0.0 msedge.net
0.0.0.0 msftncsi.com
0.0.0.0 msnbot-207-46-194-33.search.msn.com
0.0.0.0 msnbot-65-55-108-23.search.msn.com
0.0.0.0 msntest.serving-sys.com
0.0.0.0 nexus.officeapps.live.com
0.0.0.0 nexusrules.officeapps.live.com
0.0.0.0 oca.telemetry.microsoft.com
0.0.0.0 oca.telemetry.microsoft.com.nsatc.net
0.0.0.0 officeclient.microsoft.com
0.0.0.0 onesettings-bn2.metron.live.com.nsatc.net
0.0.0.0 onesettings-cy2.metron.live.com.nsatc.net
0.0.0.0 onesettings-db5.metron.live.com.nsatc.net
0.0.0.0 onesettings-hk2.metron.live.com.nsatc.net
0.0.0.0 pre.footprintpredict.com
0.0.0.0 preview.msn.com
0.0.0.0 pricelist.skype.com
0.0.0.0 rad.live.com
0.0.0.0 rad.msn.com
0.0.0.0 redir.metaservices.microsoft.com
0.0.0.0 reports.wes.df.telemetry.microsoft.com
0.0.0.0 s0.2mdn.net
0.0.0.0 sO.2mdn.net
0.0.0.0 schemas.microsoft.akadns.net
0.0.0.0 schemas.microsoft.akadns.net 
0.0.0.0 secure.adnxs.com
0.0.0.0 secure.flashtalking.com
0.0.0.0 services.wes.df.telemetry.microsoft.com
0.0.0.0 settings-sandbox.data.microsoft.com
0.0.0.0 settings-win.data.microsoft.com
0.0.0.0 settings.data.glbdns2.microsoft.com
0.0.0.0 settings.data.microsoft.com
0.0.0.0 sls.update.microsoft.com.akadns.net
0.0.0.0 sn3301-e.1drv.com
0.0.0.0 sn3301-g.1drv.com
0.0.0.0 spynet2.microsoft.com
0.0.0.0 spynetalt.microsoft.com
0.0.0.0 sqm.df.telemetry.microsoft.com
0.0.0.0 sqm.telemetry.microsoft.com
0.0.0.0 sqm.telemetry.microsoft.com.nsatc.net
0.0.0.0 ssw.live.com
0.0.0.0 static.2mdn.net
0.0.0.0 statsfe1.ws.microsoft.com
0.0.0.0 statsfe2.update.microsoft.com.akadns.net
0.0.0.0 statsfe2.ws.microsoft.com
0.0.0.0 storecatalogrevocation.storequality.microsoft.com
0.0.0.0 survey.watson.microsoft.com
0.0.0.0 t0.ssl.ak.dynamic.tiles.virtualearth.net
0.0.0.0 t0.ssl.ak.tiles.virtualearth.net
0.0.0.0 telecommand.telemetry.microsoft.com
0.0.0.0 telecommand.telemetry.microsoft.com.nsatc.net
0.0.0.0 telemetry.appex.bing.net
0.0.0.0 telemetry.microsoft.com
0.0.0.0 telemetry.urs.microsoft.com
0.0.0.0 tsfe.trafficshaping.dsp.mp.microsoft.com
0.0.0.0 ui.skype.com
0.0.0.0 v10.vortex-win.data.metron.live.com.nsatc.net
0.0.0.0 v10.vortex-win.data.microsoft.com
0.0.0.0 version.hybrid.api.here.com
0.0.0.0 view.atdmt.com
0.0.0.0 vortex-bn2.metron.live.com.nsatc.net
0.0.0.0 vortex-cy2.metron.live.com.nsatc.net
0.0.0.0 vortex-db5.metron.live.com.nsatc.net
0.0.0.0 vortex-hk2.metron.live.com.nsatc.net
0.0.0.0 vortex-sandbox.data.microsoft.com
0.0.0.0 vortex-win.data.metron.live.com.nsatc.net
0.0.0.0 vortex-win.data.microsoft.com
0.0.0.0 vortex.data.glbdns2.microsoft.com
0.0.0.0 vortex.data.metron.live.com.nsatc.net
0.0.0.0 vortex.data.microsoft.com
0.0.0.0 watson.live.com
0.0.0.0 watson.microsoft.com
0.0.0.0 watson.ppe.telemetry.microsoft.com
0.0.0.0 watson.telemetry.microsoft.com
0.0.0.0 watson.telemetry.microsoft.com.nsatc.net
0.0.0.0 web.vortex.data.microsoft.com
0.0.0.0 wes.df.telemetry.microsoft.com
0.0.0.0 win10.ipv6.microsoft.com
0.0.0.0 www.msedge.net
0.0.0.0 www.msftncsi.com

I don't know what to do with EXTRA (some domains are included in above list already)

0.0.0.0 apps.skype.com
0.0.0.0 cdn.content.prod.cms.msn.com
0.0.0.0 choice.microsoft.com.nstac.net
0.0.0.0 client.wns.windows.com
0.0.0.0 client-s.gateway.messenger.live.com
0.0.0.0 clientconfig.passport.net
0.0.0.0 deploy.static.akamaitechnologies.com
0.0.0.0 device.auth.xboxlive.com
0.0.0.0 dmd.metaservices.microsoft.com
0.0.0.0 dns.msftncsi.com
0.0.0.0 g.live.com
0.0.0.0 img-s-msn-com.akamaized.net
0.0.0.0 licensing.mp.microsoft.com
0.0.0.0 login.live.com
0.0.0.0 msftncsi.com
0.0.0.0 oneclient.sfx.ms
0.0.0.0 pricelist.skype.com
0.0.0.0 s.gateway.messenger.live.com
0.0.0.0 s0.2mdn.net
0.0.0.0 sO.2mdn.net
0.0.0.0 search.msn.com
0.0.0.0 settings-ssl.xboxlive.com
0.0.0.0 static.2mdn.net
0.0.0.0 store-images.s-microsoft.com
0.0.0.0 storeedgefd.dsx.mp.microsoft.com
0.0.0.0 tile-service.weather.microsoft.com
0.0.0.0 time.windows.com
0.0.0.0 tk2.plt.msn.com
0.0.0.0 ui.skype.com
0.0.0.0 www.msftncsi.com

and Windows Update domains:

0.0.0.0 au.download.windowsupdate.com
0.0.0.0 au.v4.download.windowsupdate.com
0.0.0.0 ctldl.windowsupdate.com
0.0.0.0 displaycatalog.mp.microsoft.com
0.0.0.0 dl.delivery.mp.microsoft.com
0.0.0.0 download.windowsupdate.com
0.0.0.0 emdl.ws.microsoft.com
0.0.0.0 fe2.update.microsoft.com
0.0.0.0 fe2.update.microsoft.com.akadns.net
0.0.0.0 fe3.delivery.dsp.mp.microsoft.com.nsatc.net
0.0.0.0 fe3.delivery.mp.microsoft.com
0.0.0.0 fg.v4.download.windowsupdate.com
0.0.0.0 microsoftwindowsupdate.net
0.0.0.0 sls.update.microsoft.com
0.0.0.0 sls.update.microsoft.com.akadns.net
0.0.0.0 statsfe2.update.microsoft.com.akadns.net
0.0.0.0 statsfe1.ws.microsoft.com
0.0.0.0 statsfe2.ws.microsoft.com
0.0.0.0 tlu.dl.delivery.mp.microsoft.com
0.0.0.0 v4.download.windowsupdate.com
0.0.0.0 windowsupdate.com
0.0.0.0 windowupdate.org

[StevenBlack]

Well let's never block Windows Update domains. That would not be good. :-)

[FadeMind]

@StevenBlack Done, see reference above.

[StevenBlack]

@FadeMind I'm having trouble picturing what's happening here. All the domains listed above ^^^ is there a reason why those are there? Maybe delete the comments because I can't find a reason why they are here, other than work in process copy-and-paste?

[StevenBlack]

@fademind Tomasz I trust your judgment here. Just tell me when I should hit the "Merge" button 😃

[FadeMind]

@StevenBlack I made 2 tasks:

1. I merged hosts from https://github.com/crazy-max/WindowsSpyBlocker with hosts extras Telemetry
2. I sorted out alphabetically hosts contents.

You can refresh hosts against lastest changes in hosts.extras.

[StevenBlack]

So... did you know... as of commits 6e01f387756ac7e187f5b0ddb73c77b09b9f886c and 43c8ced8dc032bb478cd2d2489f78d25af37a953 we can now have multiple subfolders below any data/ or extensions/ subfolder.

Like the porn extras folder currently.

Therefore it's possible to simply take one or more Windows Spy Blocker offerings, give 'em each a subfolder and update.json file and we would benefit from future new additions by the folks at Windows Spy Blocker.

This all works because we now use the glob(... /**/...) feature in Python 3.5+, and os.walk() for versions prior to Python 3..5.

[FadeMind]

Ok, I will just create separated commit about this (as extension). But Mainly is care about Telemetry (merged) for all Windows flavours. @StevenBlack do you agree?

[StevenBlack]

Yup, that's good!

But it's good to know, no need to worry about duplicates since the merge process takes care of them.

[oznu]

It looks like the "extra" content from https://github.com/crazy-max/WindowsSpyBlocker was added to the blacklist. This has had the side effect of blocking access to the Microsoft Login service, login.live.com, and preventing the skype apps from working.

[StevenBlack]

Good catch @oznu.

login.live.com comes from two sources, both new.

data/WindowsSpyBlocker/extra-win10/hosts
data/WindowsSpyBlocker/extra-win81/hosts

Scrubbing those sources now.

[StevenBlack]

Fixed! Thanks @oznu for the heads-up.

@FadeMind I renamed hosts to _hosts and update.json to _update.json in those two WindowsSpyBlocker folders.

[StevenBlack]

Issue opened at WindowsSpyBlocker.

[StevenBlack]

Based on this reply from @crazy-max, let's not include the domains in the extra-winxx files.

[MrSourceUnknown]

Maybe one of you can explain why the following Microsoft Domains are being blocked since this source started to be merged:

choice.microsoft.com < Necessary to access some Microsoft Account management, like opting out of personalised ads. answers.microsoft.com < Community Q&A forum for Microsoft related problem solving.

I can understand some specific subdomains being blocked, but those two are pretty much harmless and I don't understand how they ended up in that list in the first place.

Seems to me like this SpyBlocker content shouldn't be added to the default Unified Hosts file as long as it contains entries like this that block harmless sites/services.

[StevenBlack]

Thanks everybody. I've removed WindowsSpyBlocker from the repo.

Closing.

[crazy-max]

@MrSourceUnknown

choice.microsoft.com : It was not completely moved to the extra rules (only nsatc.net domain was there). Now this domain will be only in the extra rules. answers.microsoft.com : Based on some mitigation, this rule has been moved in the extra rules.

If you have other suggestions you can post an issue on the official repo : https://github.com/crazy-max/WindowsSpyBlocker

Thanks