Some suggested additions for your host file

[jamesleib4764]

I have some suggested additions for your host file. None cause any over-blocking for me.

cdn.permutive.app esi.edge.permutive.app

ssc-cms.33across.com

px.adentifi.com

acdn.adnxs-simple.com crcdn01.adnxs-simple.com

admp-tc-rossel.adtlgc.com

prod-m-node-1113.ssp.advertising.com

6.at.atwola.com

pod.bfmio.com

a3153.casalemedia.com

t.us1.dyntrk.com

i6.liadm.com idx.liadm.com

i4pooa2c.micpn.com qztqzde1.micpn.com

t.pubmatic.com vpaid.pubmatic.com

otheve.beacon.qq.com oth.str.beacon.qq.com cm.l.qq.com

analytics.rlcdn.com check.analytics.rlcdn.com

sync.serverbid.com

lux.speedcurve.com

fourier.taobao.com

aniview.technoratimedia.com insticator.technoratimedia.com vidcrunch.technoratimedia.com warnerbros.technoratimedia.com

sync.tidaltv.com

widgets.tree.com

73aj2-r62uz.ads.tremorhub.com m7hdh-spprw.ads.tremorhub.com pi979-10rsz.ads.tremorhub.com sml5f-2vel5.ads.tremorhub.com eq97f.publishers.tremorhub.com i9slw.publishers.tremorhub.com

collector-3309.tvsquared.com collector-6651.tvsquared.com collector-16238.us.tvsquared.com collector-9267.us.tvsquared.com

ml-client-configs.vidora.com

service.idsync.analytics.yahoo.com tag.idsync.analytics.yahoo.com

b1-nydc1.zemanta.com b1t-nydc1.zemanta.com p1.zemanta.com widgets.zemanta.com

hb.districtm.io

v.mdhv.io

pages-stats.rbl.ms res.rbl.ms

carbon-cdn.ccgateway.net eus-api.ccgateway.net pogo.ccgateway.net revenue-mapping-api.ccgateway.net script-api.ccgateway.net

ct.contentsquare.net

sync.e-planning.net

displayf-tm.everesttech.net rtb-lb-event-dx-adcloud-prod-us-east-1-tm.everesttech.net statsf-tm.everesttech.net

mpd.mxptint.net mpp.mxptint.net

rs.mail.ru

p.cpx.to

staging-pi.ispot.tv staging-ps.ispot.tv staging-pt.ispot.tv

pixel.videohub.tv

[jawz101]

where did these come from?

[jamesleib4764]

I will try to describe my setup. I use some browser extensions to block (e.g. Privacy Badger) but I want to always block as much as possible (without any overblocking) using my SquidGuard (squidguard.org) configuration. I have experimented with many block lists found on the web. The highest quality list I've found is https://github.com/StevenBlack/hosts (which includes AdAway hosts). I also wrote a shell script that creates a block list from Adblock lists like easylist and easy privacy. (It too a while to correctly handle all the @@|| unblock statements. Some lists are brutal and will over-block unless I remove some items. I learn about which items to remove by reading @@|| unblock statements, the Pi-hole Faq, Privacy Badger's "yellowlist", and trial-and error. SquidGuard is a domain-based system so the entry doubleclick.net will also block ff.doubleclick.net . I start with a list if 194874 hostnames. I keep a database of removals. I end up with list of 194002 host names and this modified list is actually quite weak because there are websites out there that fail unless they can fetch an item from the notorious pixel.adsafeprotected.com. Squid and SquidGuard have good logging facilities. I keep a database of all hosts I access (8873 entries) and all hosts that were blocked by SquidGuard (1615 entries), The hosts I listed I actually encountered. I doubt I will test the 194002 hosts in my lifetime. My porn list contains 2.5 million entries! I will annotate by list. PB-red means my Privacy Badger testing profile labels the domain as a tracker (it probably serves ads too) so unless someone wishes to directly visit fls.doubleclick.net, it's probably best to block it. sblack stands for https://github.com/StevenBlack/hosts. EasyList, EasyPrivacy, and AdGuard are easy to find. I think they are also on github.com.

Here is my list with annotations: cdn.permutive.app #permutive.app is PB-red so Privacy Badger will block this third-party domain. EasyPrivacy has ||permutive.app^ without any relevant @@|| unblock esi.edge.permutive.app #PB-red. EasyPrivacy has ||permutive.app^ without any relevant @@|| unblock

ssc-cms.33across.com #PB-red.sblack blocks parent domain 33across.com. EasyList has ||33across.com^$third-party without any relevant @@|| unblock

px.adentifi.com #PB-red. EasyList has ||adentifi.com^$third-party without any relevant @@|| unblock

acdn.adnxs-simple.com #sblack blocks parent domain adnxs-simple.com. EasyList has ||adnxs-simple.com^ without any relevant @@|| unblock crcdn01.adnxs-simple.com #sblack blocks parent domain adnxs-simple.com. EasyList has ||adnxs-simple.com^ without any relevant @@|| unblock

admp-tc-rossel.adtlgc.com #PB-red. sblack blocks parent domain adtlgc.com. EasyPrivacy has ||adtlgc.com^$third-party without any relevant @@|| unblock

[I unblock ads.adaptv.advertising.com because of @||ads.adaptv.advertising.com/einthusan.tv$xmlhttprequest,domain=imasdk.googleapis.comand] [I unblock guce.advertising.com because of @@||guce.advertising.com/collectIdentifiers$domain=guce.advertising.com] prod-m-node-1113.ssp.advertising.com #PB-red. sblack blocks parent domain advertising.com. AdGuard Mobile Ads filter has ||advertising.com^

6.at.atwola.com. sblack blocks parent domain at.atwola.com. EasyList has ||atwola.com^ without any relevant @@|| unblock

pod.bfmio.com #PB-red. sblack blocks parent domain bfmio.com. EasyPrivacy has ||bfmio.com^ without any relevant @@|| unblock

a3153.casalemedia.com #PB-red. sblack blocks parent domain casalemedia.com. AdGuard Mobile Ads filter has ||casalemedia.com^ without any relevant @@|| unblock

t.us1.dyntrk.com #PB-red. sblack blocks parent domain dyntrk.com. Easy privacy has ||dyntrk.com^$third-party without any relevant @@|| unblock

i6.liadm.com #PB-red. sblack blocks parent domain liadm.com. EasyPrivacy has ||liadm.com^ without any relevant @@|| unblock idx.liadm.com #PB-red. sblack blocks parent domain liadm.com. EasyPrivacy has ||liadm.com^ without any relevant @@|| unblock

i4pooa2c.micpn.com #PB-red. sblack blocks parent domain micpn.com. EasyPrivacy has ||micpn.com^$third-party without any relevant @@|| unblock qztqzde1.micpn.com #PB-red. sblack blocks parent domain micpn.com. EasyPrivacy has ||micpn.com^$third-party without any relevant @@|| unblock

t.pubmatic.com #PB-red. sblack blocks parent domain pubmatic.com. AdGuard Mobile Ads filter has ||pubmatic.com^ without any relevant @@|| unblock vpaid.pubmatic.com #PB-red. sblack blocks parent domain pubmatic.com AdGuard Mobile Ads filter has ||pubmatic.com^ without any relevant @@|| unblock

[I unblock pingjs.qq.com due to @@||pingjs.qq.com/h5/stats.js$script,domain=ushareit.com] otheve.beacon.qq.com #PB-red but qq.com is a useful website. sblack blocks parent domain beacon.qq.com. AdGuard Tracking Protection filter blocks parent with ||beacon.qq.com^ without any @@|| relevant unblock oth.str.beacon.qq.com #PB-red but qq.com is a useful website. sblack blocks parent domain beacon.qq.com AdGuard Tracking Protection filter blocks parent with ||beacon.qq.com^ without any @@|| relevant unblock cm.l.qq.com #PB-red but qq.com is a useful website. sblack blocks parent domain l.qq.com

analytics.rlcdn.com #PB-red. sblack blocks parent domain rlcdn.com. EasyPrivacy has ||rlcdn.com^ without any @@|| relevant unblock check.analytics.rlcdn.com #PB-red. sblack blocks parent domain rlcdn.com. EasyPrivacy has ||rlcdn.com^ without any @@|| relevant unblock

sync.serverbid.com #PB-red. EasyList has ||serverbid.com^ without any @@|| relevant unblock

lux.speedcurve.com #PB-red. sblack blocks parent domain speedcurve.com

fourier.taobao.com #PB-red but taobao.com is a useful website. AdGuard Tracking Protection filter has ||fourier.taobao.com^ without any @@|| relevant unblock

aniview.technoratimedia.com #PB-red. EasyList blocks this with ||technoratimedia.com^ without any @@|| relevant unblock insticator.technoratimedia.com #PB-red. EasyList blocks this with ||technoratimedia.com^ without any @@|| relevant unblock vidcrunch.technoratimedia.com #PB-red. EasyList blocks this with ||technoratimedia.com^ without any @@|| relevant unblock warnerbros.technoratimedia.com #PB-red. EasyList blocks this with ||technoratimedia.com^ without any @@|| relevant unblock

sync.tidaltv.com #PB-red. sblack blocks parent tidaltv.com. AdGuard Mobile Ads filter has ||tidaltv.com^ without any relevant @@|| unblock

widgets.tree.com #EasyList has ||widgets.tree.com^ without any relevant @@|| unblock

73aj2-r62uz.ads.tremorhub.com #PB-red. sblack blocks parent domain tremorhub.com. EasyList has ||tremorhub.com^ without any relevant @@|| unblock m7hdh-spprw.ads.tremorhub.com #PB-red. sblack blocks parent domain tremorhub.com EasyList has ||tremorhub.com^ without any relevant @@|| unblock pi979-10rsz.ads.tremorhub.com #PB-red. sblack blocks parent domain tremorhub.com EasyList has ||tremorhub.com^ without any relevant @@|| unblock sml5f-2vel5.ads.tremorhub.com #PB-red. sblack blocks parent domain tremorhub.com EasyList has ||tremorhub.com^ without any relevant @@|| unblock eq97f.publishers.tremorhub.com #PB-red. sblack blocks parent domain tremorhub.com EasyList has ||tremorhub.com^ without any relevant @@|| unblock i9slw.publishers.tremorhub.com #PB-red. sblack blocks parent domain tremorhub.com EasyList has ||tremorhub.com^ without any relevant @@|| unblock

collector-3309.tvsquared.com #PB-red. sblack blocks parent domain tvsquared.com collector-6651.tvsquared.com #PB-red sblack blocks parent domain tvsquared.com collector-16238.us.tvsquared.com #PB-red sblack blocks parent domain tvsquared.com collector-9267.us.tvsquared.com #PB-red sblack blocks parent domain tvsquared.com

ml-client-configs.vidora.com #sblack blocks vidora.com. EasyList has ||vidora.com^$third-party without any relevant @@|| unblock

service.idsync.analytics.yahoo.com. sblack blocks analytics.yahoo.com. AdGuard Tracking Protection filter has ||analytics.yahoo.com^ without any relevant @@|| unblock. tag.idsync.analytics.yahoo.com sblack blocks analytics.yahoo.com. AdGuard Tracking Protection filter has ||analytics.yahoo.com^ without any relevant @@|| unblock.

b1-nydc1.zemanta.com #PB-red. sblack blocks parent domain zemanta.com. EasyList has ||zemanta.com^$third-party without any relevant @@|| unblock. b1t-nydc1.zemanta.com #PB-red sblack blocks parent domain zemanta.com EasyList has ||zemanta.com^$third-party without any relevant @@|| unblock. p1.zemanta.com #PB-red sblack blocks parent domain zemanta.com EasyList has ||zemanta.com^$third-party without any relevant @@|| unblock. widgets.zemanta.com #PB-red sblack blocks parent domain zemanta.com EasyList has ||zemanta.com^$third-party without any relevant @@|| unblock.

hb.districtm.io #PB-red. sblack blocks parent domain districtm.io. EasyList has ||districtm.io^ without any relevant @@|| unblock.

v.mdhv.io #PB-red. EasyPrivacy has ||mdhv.io^ without any relevant @@|| unblock.

pages-stats.rbl.ms #AdGuard Tracking Protection filter has ||pages-stats.rbl.ms^$third-party without any relevant @@|| unblock. res.rbl.ms #AdGuard Tracking Protection filter has ||res.rbl.ms^$third-party without any relevant @@|| unblock.

carbon-cdn.ccgateway.net #PB-red. sblack blocks parent domain ccgateway.net. EasyPrivacy and AdGuard Tracking Protection filter have ||ccgateway.net^$third-party without any relevant @@|| unblock. eus-api.ccgateway.net #PB-red. sblack blocks parent domain ccgateway.net. EasyPrivacy and AdGuard Tracking Protection filter have ||ccgateway.net^$third-party without any relevant @@|| unblock. pogo.ccgateway.net #PB-red. sblack blocks parent domain ccgateway.net. EasyPrivacy and AdGuard Tracking Protection filter have ||ccgateway.net^$third-party without any relevant @@|| unblock. revenue-mapping-api.ccgateway.net #PB-red. sblack blocks parent domain ccgateway.net. EasyPrivacy and AdGuard Tracking Protection filter have ||ccgateway.net^$third-party without any relevant @@|| unblock. script-api.ccgateway.net #PB-red. sblack blocks parent domain ccgateway.net. EasyPrivacy and AdGuard Tracking Protection filter have ||ccgateway.net^$third-party without any relevant @@|| unblock.

ct.contentsquare.net #PB-red. sblack blocks parent domain contentsquare.net. EasyPrivacy and AdGuard Tracking Protection filter have ||contentsquare.net^$third-party without any relevant @@|| unblock.

sync.e-planning.net #PB-red, sblack blocks parent domain e-planning.net. EasyList has ||e-planning.net^$third-party without any relevant @@|| unblock.

displayf-tm.everesttech.net #PB-red. sblack blocks parent domain everesttech.net. EasyPrivacy has ||everesttech.net^ without any relevant @@|| unblock. rtb-lb-event-dx-adcloud-prod-us-east-1-tm.everesttech.net #PB-red. sblack blocks parent domain everesttech.net. EasyPrivacy has ||everesttech.net^ without any relevant @@|| unblock. statsf-tm.everesttech.net #PB-red. sblack blocks parent domain everesttech.net. EasyPrivacy has ||everesttech.net^ without any relevant @@|| unblock.

mpd.mxptint.net #PB-red. sblack blocks parent domain mxptint.net. EasyPrivacy has ||mxptint.net^$third-party without any relevant @@|| unblock. mpp.mxptint.net #PB-red. sblack blocks parent domain mxptint.net. EasyPrivacy has ||mxptint.net^$third-party without any relevant @@|| unblock.

[I unblock ad.mail.ru because of @@||ad.mail.ru/adp/?q=*&target_count=$domain=youla.ru and @@||ad.mail.ru/adq/?callback=jQuery$domain=target.my.com] [I unblock top.mail.ru because ||top.mail.ru^$badfilter and it is only partially blocked by ||top.mail.ru/jump$third-party] rs.mail.ru #EasyPrivacy has ||rs.mail.ru^ without any relevant @@|| unblock.

p.cpx.to #PB-red. sblack blocks parent domain cpx.to. EasyPrivacy has ||cpx.to^ without any relevant @@|| unblock.

staging-pi.ispot.tv #PB-red. sblack blocks parent domain ispot.tv staging-ps.ispot.tv #PB-red. sblack blocks parent domain ispot.tv staging-pt.ispot.tv #PB-red. sblack blocks parent domain ispot.tv

pixel.videohub.tv #PB-red. sblack blocks parent domain videohub.tv. EasyPrivacy has ||pixel.videohub.tv^ without any relevant @@|| unblock.

[jawz101]

Well the AdAway list is intended to target observed, mobile-app specific ad and tracking companies (I.e they have an sdk developers will incorporate into their app and to monetize users.). I just want to keep the AdAway list limited to that niche because it's a niche that was severely unmoderated.

Ex: Inmobi does ads in mobile apps.

https://www.inmobi.com/

https://support.aerserv.com/hc/en-us/articles/204159160-Android-SDK-Integration-Version-3-1-8

A few years ago you'd never see it in blocklists because list makers just look at things in uBlock Origin or other website ad blockers.

I just want to keep the list focused on that primary intention. I recognize most of the entries you've provided just because they look like your typical 3rd party company names. But the things I do to keep the list tight is observe a mobile app (excluding web browsers) do a name lookup, I'll look at the company's website, support documentation, case studies to see their primary business. I'll also check the domains on VirusTotal /relations/communicating files to see if their sandbox tests show that a domain is contacted by Android APK analysis.

And then every now and then I download a few log files from the daily Cisco Top1 million DNS lookups to see if entries are still relevant because it's an industry that's constantly moving and I'd rather not keep the leftovers.

[jamesleib4764]

Thanks for the clarification. I may have assumed that your list was "general purpose". Worrying about requests that will never happen will only bloat your list. Good luck with your project!