search

lightswitch05 / hosts

Hostfile blocklist for ads and tracking, updated regularly
https://www.github.developerdan.com/hosts/
Apache License 2.0
1.52k stars 76 forks source link

[False Positive] Xbox Live Achievement Domains blocked #161

Closed Bryantdl7 closed 3 years ago

Bryantdl7 commented 4 years ago

Thank you for using my hosts lists. I appreciate feedback on this project, but I do expect you to put effort into your request. At the end of the day, this is my list and I maintain it for my usage. I'll address reasonable requests, but I cannot make everyone happy. This means you might have to whitelist a few things on your own. A list that breaks nothing is a list that blocks nothing.

The second I unblocked these domains, the xboxes began notifying me of achievements from months ago (I have been using your list for a few months now)

Here's the domains I have it narrowed down to, I didn't check the list to see which ones arent present on your https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt list.

Here are the domains, this has been crazy to track down!:

v20.events.data.microsoft.com - on your list watson.telemetry.microsoft.com - on your list web.vortex.data.microsoft.com - on your list v10.events.data.microsoft.com - on your list

========================================= Domains not on your list, but may help some sorry soul having this problem in the future DuckDuckGo'ing the solution. Make sure all these domains are allowed if you use multiple lists:

attestation.xboxlive.comcert.mgt.xboxlive.com ctldl.windowsupdate.comdef-vef.xboxlive.com device.auth.xboxlive.comeds.xboxlive.com help.ui.xboxlive.comlicensing.xboxlive.commicrosoft.com notify.xboxlive.comsettings-win.data.microsoft.com title.auth.xboxlive.comtitle.mgt.xboxlive.com v10.vortex-win.data.microsoft.com www.msftncsi.com xbox.ipv6.microsoft.com xboxexperiencesprod.experimentation.xboxlive.com xflight.xboxlive.comxkms.xbolive.com xsts.auth.xboxlive.com v20.events.data.microsoft.com watson.telemetry.microsoft.com web.vortex.data.microsoft.com v10.events.data.microsoft.com

lightswitch05 commented 4 years ago

I didn't check the list to see which ones arent present

So... I’m expected to do that for an issue that your having, which may or may not have anything to do with my list because you won’t look to see if they are in there or not?

As my issue template says: I do expect you to put some effort into your request.

Bryantdl7 commented 4 years ago

I have put three months into figuring this out, sorry. I’ll comment in a moment telling you which ones are in your list.

Also, I only run your list, not a single additional list, so I am certain that when I whitelisted these domains it fixed a problem of a false positive domain in your list.

Sent from my iPhone

On Apr 16, 2020, at 7:22 PM, Daniel notifications@github.com wrote:



I didn't check the list to see which ones arent present

So... I’m expected to do that for an issue that your having, which may or may not have anything to do with my list because you won’t look to see if they are in there or not?

As my issue template says: I do expect you to put some effort into your request.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/lightswitch05/hosts/issues/161#issuecomment-614946169, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKOR5OSG4QAD7F3HOLK4ZLRM6HMDANCNFSM4MKIHNLA.

Bryantdl7 commented 4 years ago

Here are the domains present on your list, original post updated to note this as well:

v20.events.data.microsoft.com watson.telemetry.microsoft.com web.vortex.data.microsoft.com v10.events.data.microsoft.com

Most of these domains are mentioned in the following forum post: https://answers.microsoft.com/en-us/xbox/forum/all/local-cache-is-full-stats-and-achievements-are-no/222d572a-56d3-45a1-97bc-b907c007e2dd

Additionally, Watson telemetry, in the world of windows is used to upload crash logs for when things like programs (games in this case) crash. That is helpful for developers to fix problems with their games, I am unsure how this relates to the overall problem of achievements other than that, but it was part of the long journey of slowly unblocking sites denied by my Xboxes.

Xbox has an annoying feature where you can not change a DHCPD-set IPv6 DNS server to a different value. I am certain others using your list will encounter this again in the future if they use pihole in DHCP mode, with IPv6 enabled, as the Xbox One prefers IPv6 when available.

Apologies for being vague in my original post.

lightswitch05 commented 4 years ago

Thanks for the updated report @Bryantdl7 - I'm very busy at the moment, but I will look at this as soon as I have some time.

Bryantdl7 commented 4 years ago

No problem, thanks for considering this change.

Sent from my iPhone

On Apr 17, 2020, at 8:43 AM, Daniel notifications@github.com wrote:



Thanks for the updated report @Bryantdl7https://github.com/Bryantdl7 - I'm very busy at the moment, but I will look at this as soon as I have some time.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/lightswitch05/hosts/issues/161#issuecomment-615223121, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKOR5OOHUJMJHOUNVLQSDDRNBFGPANCNFSM4MKIHNLA.

XhmikosR commented 4 years ago

I'd be very careful with removing the aforementioned domains before extensive research.

Just my 2 cents.

Bryantdl7 commented 4 years ago

I agree, it’s hard to truly tell what else gets pushed to these domains, which brings up a bigger question, what services connect via static IP on Microsoft products, and completely bypass the need for DNS? Sadly not Xbox achievements use something like this, but if you run a Windows PC and check out wire shark, there is a lot more than DNS phoning home.

I am in no way condoning how all these big companies make it hard to know what they collect, I don’t even have windows installed on my computer any more. But at some point we have to accept that a network connection to the world will bring in some security concerns which can only be fixed by limiting WAN exposure.

I think it sucks how much data companies mine on us, but it never can truly be blocked 100% of the time unless you run an operating system you compile yourself, and only install bare minimum software resources that you check for security concerns ahead of time.

A better long term solution is when the new Pihole version which is in beta releases, we can manually set certain devices to not adhere to block lists, or at least that’s what I’ve heard. Microsoft has the Xbox one set up so if you receive DNS info over ipv6 it will always prioritize it, and you may not change ipv6 info yourself. This leaves me with asking a list maintainer to add it to their list fully unblocking it for every system on my network, or whitelisting it for my whole network. I guess I could also burst allow these domains by temporary disabling Pihole network-wide, as to limit my exposure, but at that point is it really worth it?

Anyways, that’s my networking/cyber security rant of the day. I’m just happy I figured out what domains are the cause of this even if they don’t get pulled off the list.

I part with this argument I alluded to earlier - if you want 100% security on a device, never connect it to the Internet. This will guarantee you the highest level of security imaginable.

Sent from my iPhone

On Apr 18, 2020, at 4:29 AM, XhmikosR notifications@github.com wrote:



I'd be very careful with removing the aforementioned domains before extensive research.

Just my 2 cents.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/lightswitch05/hosts/issues/161#issuecomment-615755495, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABKOR5P4RHVPDCGAJURO353RNFQGTANCNFSM4MKIHNLA.

lightswitch05 commented 4 years ago

I added both *.telemetry.microsoft.com and *.events.data.microsoft.com on 2019-11-06 in https://github.com/lightswitch05/hosts/commit/9dd786a8

Additionally, Watson telemetry, in the world of windows is used to upload crash logs for when things like programs (games in this case) crash. That is helpful for developers to fix problems with their games

Unfortunately diagnostic data like crash logs and tracking/analytics data is generally vacuumed up by the same tools and then separated out server side. Also, crash logs often contain sensitive information like PII. So my general rule of thumb is to block it all and let people whitelist whatever they decide is trustworthy. I prefer opt-in.

Just to see what some other list maintainers are doing, I searched the lists I have on my pihole for the given hosts, and came across these:

* v20.events.data.microsoft.com
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
* watson.telemetry.microsoft.com
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
    * https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
    * https://raw.githubusercontent.com/michaeltrimm/hosts-blocking/master/_hosts.txt
    * https://raw.githubusercontent.com/vokins/yhosts/master/hosts
    * https://hosts-file.net/ad_servers.txt
    * https://raw.githubusercontent.com/bigdargon/hostsVN/master/hosts
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
    * https://raw.githubusercontent.com/jerryn70/GoodbyeAds/master/Hosts/GoodbyeAds.txt
* web.vortex.data.microsoft.com
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/data/StevenBlack/hosts
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
    * https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt
    * https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
* v10.events.data.microsoft.com
    * https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
    * https://raw.githubusercontent.com/anudeepND/blacklist/master/adservers.txt

The results are really all over the place. Given the auto-expand nature of my list, and that I have telemetry.microsoft.com blocked - its not surprising that I have some of the better sub-domain coverage compared to some of the other lists. But it also could be some other list maintainers have decided to whitelist some subdomains. Its interesting that Steven Black's list has v10c.events.data.microsoft.com in it, but not v10.events.data.microsoft.com.

Alright... so the reason I'm showing that a bunch of other lists block these too: If I chose to move them to the aggressive list, its very likely most people will still have to whitelist them. Particularly since Steven's list is a default PiHole list.

I hate that I'm breaking Xbox for you and that it took a long time for you to figure it out. I really appreciate you sharing your results back with me. I'm always saying that I appreciate feedback and it is true. I make a lot of mistakes. In this case however, we know that Microsoft is using these hosts to do tracking, data mining, segmenting, and then targeting people with that data.

At this time I'm going to leave the list blocked. But lets leave the ticket open to make it easier for people to find. I may even add a section to the README that links here.

Bryantdl7 commented 4 years ago

Fair enough, I appreciate you putting some research into the inner-workings of these domains.

At least I know what to focus on unblocking going forward. It might be helpful like you said to add this into the readme due to the tons of people running into this that I have found on the internet.

On 4/18/20 2:48 PM, Daniel wrote:

I added both |.telemetry.microsoft.com| and |.events.data.microsoft.com| on 2019-11-06 in 9dd786a https://github.com/lightswitch05/hosts/commit/9dd786a8

Additionally, Watson telemetry, in the world of windows is used to
upload crash logs for when things like programs (games in this
case) crash. That is helpful for developers to fix problems with
their games

Unfortunately diagnostic data like crash logs and tracking/analytics data is generally vacuumed up by the same tools and then separated out server side. Also, crash logs often contain sensitive information like PII. So my general rule of thumb is to block it all and let people whitelist whatever they decide is trustworthy. I prefer opt-in.

Just to see what some other list maintainers are doing, I searched the lists I have on my pihole for the given hosts, and came across these:

| v20.events.data.microsoft.com https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt

The results are really all over the place. Given the auto-expand nature of my list, and that I have |telemetry.microsoft.com| blocked - its not surprising that I have some of the better sub-domain coverage compared to some of the other lists. But it also could be some other list maintainers have decided to whitelist some subdomains. Its interesting that Steven Black's list has |v10c.events.data.microsoft.com| in it, but not |v10.events.data.microsoft.com|.

Alright... so the reason I'm showing that a bunch of other lists block these too: If I chose to move them to the aggressive list, its very likely most people will still have to whitelist them. Particularly since Steven's list is a default PiHole list.

I hate that I'm breaking Xbox for you and that it took a long time for you to figure it out. I really appreciate you sharing your results back with me. I'm always saying that I appreciate feedback and it is true. I make a lot of mistakes. In this case however, we know that Microsoft is using these hosts to do tracking, data mining, segmenting, and then targeting people with that data.

At this time I'm going to leave the list blocked. But lets leave the ticket open to make it easier for people to find. I may even add a section to the README that links here.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lightswitch05/hosts/issues/161#issuecomment-615924919, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKOR5KHKTKFM7GEFMPBWKLRNHYWHANCNFSM4MKIHNLA.

liamengland1 commented 4 years ago

The domains are already listed on https://discourse.pi-hole.net/t/commonly-whitelisted-domains/212. How did this take you three months to figure out?

beerisgood commented 4 years ago

Just to see what some other list maintainers are doing, I searched the lists I have on my pihole for the given hosts, and came across these:

Problem is that many people only copy domains from other, without think or read about it.

It doesn't make sense to use a OS which you don't trust. Also Windows provide disable sending personal info. At minimum level, only "needed" data are send.

Of course privacy "experts" think they're better then official documentation from Microsoft and telemetry is always bad, which is totally nonsense. Sadly this would never change.

I already reporting a lot of these as false positive at different lists but i guess you know what most dev's answer. Only few make the right decision :(

lightswitch05 commented 4 years ago

Hey @beerisgood, the decision was made and I provided my line of thought. Your welcome to voice your opinion that the blocks are too aggressive, but leave it at that. There is no cause to try and belittle people, no one here is claiming to be an expert. If you do not like my list that I maintain for my own personal use then please do not use it.

@llacb47 some people have a harder time tracking down troublesome blocks then you do 😀

lightswitch05 commented 4 years ago

I added a section to the README about this: https://github.com/lightswitch05/hosts#common-issues

soredake commented 3 years ago

This also breaks authorization in "your phone" app from microsoft.