search

PowerDNS / pdns

PowerDNS Authoritative, PowerDNS Recursor, dnsdist
https://www.powerdns.com/
GNU General Public License v2.0
3.71k stars 908 forks source link

Duplicate entries in RPZ files should be ignored #12842

Closed ruffy91 closed 12 months ago

ruffy91 commented 1 year ago

Short description

Loading a RPZ file with duplicate entries should not prevent loading that file

Usecase

I use the ThreatFox RPZ zone file from abuse.ch to block C&C domains. The file sometimes contains duplicate entries.

Description

Loading a RPZ file containing duplicate entries should not fail with msg="Exception while loading RPZ zone from file" error="Adding a QName-based filter policy of kind . but a policy of kind . already exists for the following name: soul-kissed.org" subsystem="luaconfig" level="0" prio="Error" tid="0" ts="1684447502.277" file="/etc/powerdns/threatfox.rpz It should instead ignore the duplicate entries, write a warning and finish loading the file

Habbie commented 1 year ago

The file sometimes contains duplicate entries.

without judging your request as valid or invalid, have you asked them to generate less invalid zone files?

ruffy91 commented 1 year ago

I considered it, however the zonefile is a valid RFC1035 zonefile and according to the ietf draft for RPZs https://www.ietf.org/archive/id/draft-vixie-dnsop-dns-rpz-00.txt the precedence should be used which means only the first duplicate RRset should be used (5.2. "RPZ Ordering" Precedence Rule).

Or if you consider it invalid for RPZ use the RRset should be ignored and an error logged ("If an RR found in an RPZ is meaningless or unusable for response policy purposes, then the containing RRset SHOULD be ignored, and an error MAY be logged.").

Now that you asked I also sent an email to the sole contributor of abuse.ch asking to remove duplicate entries from the RPZ in the future However I still think that duplicate entries are not invalid.

omoerbeek commented 1 year ago

I considered it, however the zonefile is a valid RFC1035 zonefile and according to the ietf draft for RPZs https://www.ietf.org/archive/id/draft-vixie-dnsop-dns-rpz-00.txt the precedence should be used which means only the first duplicate RRset should be used (5.2. "RPZ Ordering" Precedence Rule).

First line of that section: This precedence rule applies when the matches being compared refer to policy rules in different RPZs. It does not say anything about rules within an RPZ.

But the second quote suggests we should be less strict indeed.

i8degrees commented 10 months ago

I do not wish to re-open this issue, but to say thank you so much for patching this issue. I was on the verge of abandoning the pdns recursor entirely until I discovered this issue and realized that my configuration was solid and bug free.

I use several RPZ lists from ioc2rpz on top of my own collection of domains to either drop or pass through. I don't have (direct, anyways!) control over the ~1,000,000 domains that come from that list and the effort that it would have taken me to pre-process these lists just to ensure that there are no collisions between my personal list and the rest would just not have been worth it at all. I would have had to settle with PiHole. (Nothing against that, just not what best suits my network).

As a bonus, I noticed that memory consumption decreased dramatically for me, probably due to another issue filed that mentions memory use with RPZ.

Now I only have to sit tight and patiently await for the v5 release of the recursor to be officially released...

Thank you once again.