Just a question... - Githubissues

badmojr / 1Hosts

World's most advanced DNS filter-/blocklists!

https://o0.pages.dev

Mozilla Public License 2.0

1.56k stars 90 forks source link

Just a question... #234

Closed crssi closed 3 years ago

crssi commented 3 years ago

I have noticed that in your assets the OISD is also listed.

How do you decide what goes in (for example Pro) your collection and what not.

For example:

surveystats.hotjar.io
publisher-assets.spot.im
static.klaviyo.com
alpaca.reclaimthenet.org
login.live-int.com
widgets.pinterest.com
p3a.brave.com
chimpstatic.com
metrics.washingtonpost.com
widget-mediator.zopim.com
api.ipgeolocation.io
geo.query.yahoo.com
middycdn-a.akamaihd.net
wh.vecer.com
aaa.bisnode.si
bigduckgames.s3.amazonaws.com

I don't expect that you would analyze the example listing... its just an example. I am just curious how those didn't make to the Pro list (or any other) from OISD asset.

Thank you, Cheers

badmojr commented 3 years ago

Hello! For certain lists such as OISD:-

they get combined with the available exclusions/false positives list(s).
Then, the resultant combined list gets further broken down; by ditching subdomains if the parent/root domain of the said subdomains is present in the list.
I then proceed to remove the domains in the exclusions/false positives list(s) from the 'broken down' list. This step is where domains such as bigduckgames.s3.amazonaws.com, & middycdn-a.akamaihd.net get ommitted.
Lastly, the final list from above step gets matched for sub-/domains with the original OISD list.
For some big lists, their lists' contents get further matched with http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip, hence reducing the lists' size considerably.

Hope that helps clarify things!

crssi commented 3 years ago

Its funny that you mention the bigduckgames.s3.amazonaws.com, because that was exactly the one that caught my eye and made me raise this question. 😄 lol I know that the bigduckgames.s3.amazonaws.com is the ADV source for in-game ADV.... sample app is Flow Free on iOS. Why happens to be on exclusions/false positives list(s)?

Cheers

badmojr commented 3 years ago

It (exclusions/false positives list) contains entries such as: yahoo.com akamaihd.net amazonaws.com pinterest.com washingtonpost.com ipgeolocation.io spot.im etc.

In short, any domain that is whitelisted.

crssi commented 3 years ago

I see, the whitelisted are taken as a wildcard.

Thank you @badmojr ❤️

Cheers

badmojr commented 3 years ago

Hehehe!! Should have made the above as TL;DR

😄 Cheers!