Many duplicates in CNAME list

StevenBlack commented 1 year ago

Hello 👋🏻 everyone!

I've been reviewing the CNAME list at this URL:

https://raw.githubusercontent.com/AdguardTeam/cname-trackers/master/combined_disguised_trackers_justdomains.txt

When I process this list in the following way:

Remove comments
Remove blank lines

I observe: 90,826 domains.

If I then...

Sort unique

I observe: 83,546 domains.

THEREFORE it appears the list is presently carrying 7,280 duplicate lines, about 8% of all lines.

Some domains are duplicates, but others have many copies. For example, the domain starget.mathworks.com appears 10 times.

This is an easy fix, something to add to your regular process?

Keep up the good work!

Alex-302 commented 1 year ago

Not all records are included to filters. For example ahacdn.me was disabled(over 20k of redundant rules) https://github.com/AdguardTeam/AdguardFilters/commit/971b2149c57d8ad4e2fbd0336ce20162915fe210

hagezi commented 1 year ago

Top 50 FLD count of unique disguised CNAME Trackers - just domains

  14326 daraz.com
  11294 footprintdns.com
   7819 office.com
   4439 reminews.com
   4015 gofenews.com
   2452 guesswhatnews.com
   2382 jemonews.com
   1921 luronews.com
   1881 omtrdc.net
   1673 lootynews.com
   1141 zooplus.de
    967 zooplus.it
    577 deeginews.com
    576 actonservice.com
    482 sifenews.com
    430 zooplus.fr
    361 notifysrv.com
    338 rookinews.com
    330 huronews.com
    328 shockynews.com
    310 tegronews.com
    289 zooplus.nl
    217 loopanews.com
    200 zooplus.com
    179 tns-gallup.dk
    167 iocnt.de
    159 notify6.com
    131 zooplus.es
    101 vgwort.de
     75 selornews.com
     31 edu.au
     29 ne.jp
     27 skf.com
     25 espmp-nifr.net
     24 samsung.com
     22 optimizesrv.com
     19 sfr.fr
     19 espmp-agfr.net
     18 americanexpress.com
     17 microsoft.com
     16 or.jp
     16 fupa.net
     16 ensighten.com
     15 intuit.com
     15 focus.de
     15 b2bmktg.com
     14 healthgrades.com
     14 espmp-aufr.net
     13 zettle.com
     13 younited-credit.com

Alex-302 commented 1 year ago

It is hard to separate ad/tracking and non-ad domains. For example all *.net.daraz.com subdomain rules cannot be replaced by blocking of net.daraz.com

You can just use our DNS, and that domains will be blocked(regular filters don't matter in this case).

Details

![image](https://user-images.githubusercontent.com/8361299/202199536-1165fbf7-0c9e-4431-9053-2038b0bc26e9.png)

krystian3w commented 1 year ago

See this (8199 lines can by removed in post-production aka network bandwidth limits (+116630 B, 11.35 kibibytes)):

COUNT    | LINE
-----------------------------------------------------
      15 | abt.nike.com
      15 | at.neom.com
      15 | atarget.harley-davidson.com
      15 | satgt.grafana.com
      15 | sats.spiegel.de
      15 | securemvt.apple.com
      15 | starget.aircanada.com
      15 | starget.bitdefender.com
      15 | starget.collegeboard.org
      15 | starget.intel.com
      15 | starget.mathworks.com
      15 | stt.dell.com
      15 | stt.nvidia.com
      15 | target-us.samsung.com
      15 | target.accenture.com
      15 | target.bankofamerica.com
      15 | target.bose.com
      15 | target.cisco.com
      15 | target.footlocker.com
      15 | target.microsoft.com
      15 | target.nejm.org
      15 | target.nxp.com
      15 | targetlr.adobe.com
      15 | tgt.maep.ibm.com
      15 | tmetrics.hdfcbank.com
      15 | webtarget.logitech.com
      15 | webtarget.logitechg.com
     (...)
-----------------------------------------------------
   63429 | TOTAL LINES
    6527 | DISTINCT DUPLICATE LINES

anti CPU slowdown: ```adb github.com###user-content-info-repeat github.com###user-content-repeat-counted:remove() ```` more under spoiler:

2-13 repeating (descending)

AdguardTeam / cname-trackers

Many duplicates in CNAME list #61