Add duplicate and redundancy checking

[Alex-302]

The size of some lists is greatly increased due to the presence of redundant rules. We need to check rules for redundancy and remove unnecessary ones.

For example:

||tracker.example.com^
||sub.domain.tracker.example.com^
||various.subdomains.tracker.example.com^

2, 3 are redundant.

and

||www.1016.lookinews.com^
||1016.lookinews.com^

The first is redundant.

Online tool https://abpvn.com/ruleChecker/redundantRuleChecker.html

Code of this tool domain_check.zip

[ameshkov]

The easiest way would be to use tsurlfilter here.

[slavaleleka]

is it redundant for Safari browser as well?

[Alex-302]

@slavaleleka In Safari || matches only one subdomain, as I remember.

[Alex-302]

If 3.2.1.sub.domain.com: ||sub.domain.com^ - for regular ad blockers .sub.domain.com^ - addition only for Safari CB. It probably doesn't make sense to do that.

[slavaleleka]

it makes sense to be done but not in this repo then

[jellizaveta]

The task was moved to the appropriate repositories. https://github.com/AdguardTeam/FiltersRegistry/issues/946 https://github.com/AdguardTeam/FiltersCompiler/issues/217