liamengland1 / mischosts

hosts for pihole
MIT License
138 stars 26 forks source link

Automatic TicTok subdomain discovery #11

Closed lightswitch05 closed 4 years ago

lightswitch05 commented 4 years ago

After seeing some TikToc domains on my network, and no one having TicToc installed, I've become interested in taking a more active role to ensure TikToc is blocked. There are a lot of regex lists out there, but unfortunately you cannot subscribe to regex lists in PiHole. I believe my automatic subdomain discovery tools could be very effective with TicTok.

Issues:

Possible Solution:

@llacb47 your hosts list is excellent, but unfortunately subdomains are getting through. Your regex list is even better, but I cannot subscribe to it. How would you feel if I added your list to my tools so that it can auto-expand the subdomains? Like I said, I have no interest in actually installing TikToc, so my ability to actually curate the list would not be that great. I'd be happy to let you manage it - say what should be added or removed. Its your list, so I don't want to manage it. I'm unable to expose my management tools publicly since I didn't design it to work like that, but I would take action on whatever additions or removals you would like to make. I believe I can modify my tools where the list would remain in your project. After all, its your list. For it to work, I would need write access. Also, the nice way your have the domains grouped with comments breaking up each section - there is no way I could continue that since the list is automatically generated. If you have any interest, you could create a branch as a test run and decide if that is something you like or not.

Anyways, I really hate seeing TicTok domains getting though, but I'm also not interested in having it manage it myself.

lightswitch05 commented 4 years ago

Oh, it turns out I was using tiktok-hosts instead of tiktok-block. However, my offer still stands for doing subdomain discovery if you are interested.

lightswitch05 commented 4 years ago

I've noticed that the tiktok-block list does not block a lot of the root domains that the regex list does. Is this on purpose?

Domain Blocked in Regex Blocked in Hosts
bytecdn.cn yes no
bytedance.com yes no
bytedance.net yes no
bytedns.net yes no
byteicdn.com yes yes
byteimg.com yes no
byteoversea.com yes yes
byteoversea.net yes no
bytetcdn.com yes no
hypstarcdn.com yes no
ibytedtos.com yes no
ibyteimg.com yes no
ipstatp.com yes yes
isnssdk.com yes no
muscdn.com yes no
musemuse.cn yes no
musical.ly yes yes
myqcloud.com yes no
ovscdns.net yes no
pstatp.com yes yes
sgpstatp.com yes no
sgsnssdk.com yes no
snssdk.com yes yes
tiktok.com yes yes
tiktokcdn.com yes no
tiktokv.com yes no
toutiao.com yes no
worldfcdn.com yes no
wsdvs.com yes no
liamengland1 commented 4 years ago

??? You're looking at the wrong list... I block all of those domains. https://github.com/llacb47/mischosts/blob/master/social/tiktok-block

One of us might be confused.

The goal of this list is to block trackers but allow the app/website to continue working: https://github.com/llacb47/mischosts/blob/master/tiktok-hosts

The goal of this list is to block tiktok entirely: https://github.com/llacb47/mischosts/blob/master/social/tiktok-block

lightswitch05 commented 4 years ago

I just double checked. You block many subdomains of tiktokcdn.com, but you do not block tiktokcdn.com itself.

liamengland1 commented 4 years ago

What would that do? It doesn't resolve anywhere. I suppose I can add it for people who use adguard home or other services that automatically block subdomains.

id 59843
opcode QUERY
rcode NOERROR
flags QR RD RA
;QUESTION
tiktokcdn.com. IN A
;ANSWER
;AUTHORITY
tiktokcdn.com. 1799 IN SOA a9-66.akam.net. hostmaster.akamai.com. 1562918823 43200 7200 604800 7200
;ADDITIONAL
lightswitch05 commented 4 years ago

That was just an example of one of them. I'm curious if there are any in the list above that are not blocked for compatibility issues or if they just aren't being used and so are excluded?

liamengland1 commented 4 years ago

Added some here: https://github.com/llacb47/mischosts/commit/8be0340cd13481b286d9896bd09bfabbe81761ed

Rationale behind the non-obvious domains:

Blocking: worldfcdn.com - seems to be only used for tiktok, ownership is not obvious https://whois.ename.net/WorldfCdn.com Some subdomains are in Bytedance's ASN and others are in a company called Zenlayer's.

Not blocking: wsdvs.com - Domain used by Chinese CDN Wangsu (网宿) - https://is.gd/MXDyVv myqcloud.com - Domain used by Tencent Cloud - https://intl.cloud.tencent.com/document/product/436/6224 ovscdns.net- Another domain used by Tencent

lightswitch05 commented 4 years ago

Thanks for the clarification!

liamengland1 commented 4 years ago

Well, seems like someone didn't like my comments above :(