search

gaenserich / hostsblock

an ad- and malware-blocking script for Linux
https://github.com/gaenserich/hostsblock
225 stars 28 forks source link

Data loss in the process of merging and sorting blocklists? #33

Closed Sadi58 closed 9 years ago

Sadi58 commented 9 years ago

I have HostsMan running in a Windows machine configured to use exactly the same sources as Linux hostsblock, but it produces considerably larger list.

The sources used are: http://winhelp2002.mvps.org/hosts.zip http://pgl.yoyo.org/as/serverlist.php?hostformat=hosts&mimetype=plaintext http://www.malwaredomainlist.com/hostslist/hosts.txt http://hosts-file.net/ad_servers.asp http://hostsfile.mine.nu/Hosts.zip http://someonewhocares.org/hosts/hosts http://sysctl.org/cameleon/hosts

When I use Meld to compare two lists (after stripping everything other than domain names) I see that there are also a number of domain names missing in the Windows hosts list, but it has 147957 entries whereas hosts.block has 127335.

For instance, I saw large chunks of entries ending in hosts.block such as:

a.collective-media.net...302br.net ad.adk2.co ad.amtk-media.com...302br.net ad.doubleclick.net...302br.net ad-emea.doubleclick.net...302br.net adfarm.mediaplex.com...302br.net admin.testandtarget.omniture.com ads.pointroll.com...302br.net ams*.ib.adnxs.com

gaenserich commented 9 years ago

What sort of entries are missing in hosts.block compared to HostsMan? I'm wondering if it might be a problem with HostsMan no doing adequate deduplication (or if hostsblock isn't doing aggressive enough pattern matching).

Re: The second part (hosts.block having entries not in the Windows hosts list): Is there a problem with those specific entries? Are there redundancies in them that didn't get filtered out?

Sadi58 commented 9 years ago

I think I've found the cause of the problem! I have a virtually empty file here: /var/cache/hostsblock/hosts-file.net.ad_servers.asp It seems this source in the hostsblock.conf file should be revised as http://hosts-file.net/ad_servers.txt

<head><title>Object moved</title></head>
<body><h1>Object Moved</h1>This object may be found <a HREF="ad_servers.txt">here</a>.</body>

As for the missing entries in the output of Windows HostMan, I wouldn't bother about it really ;-)

gaenserich commented 9 years ago

I'll adjust the config file (4e7c566cbf1e695068b7f8dc6c65843b2f2c2af6). This could also be a consequence of https://github.com/gaenserich/hostsblock/issues/27 an upstream bug in curl.

Sadi58 commented 9 years ago

Sorry, I've just noticed that the same also applies to http://hosts-file.net/hphosts-partial.asp