Drop rule for in-addr.arpa

[quanah]

Can't see a valid reason for this rule to exist:

/.in-addr.arpa$/ REJECT Generic - Please relay via ISP

[stevejenkins]

Hi, Quanah. Is there no potential for that string to show up in an EHLO?

[quanah]

Hm, looking at are logs, we get both valid and spam emails that have .in-addr.arpa as part of their address. :/ I.e., the rule itself seems overly broad, as it catches anything with that address space.

[jwpemail]

Who sends valid email with a poorly configured rDNS PTR? That's almost insane as few places would easily accept it.

Proof that the rule does work:

Nov 3 18:38:05 svr5 postfix/smtpd[17078]: NOQUEUE: reject: RCPT from unknown[200.52.74.178]: 554 5.7.1 <178.74.52.200.in-addr.arpa>: Unverified Client host rejected: Generic - Please relay via ISP; from=lisa.king@techsolve.org to=ddzevie@netcoolusers.org proto=ESMTP helo=

[stevejenkins]

Thanks, guys. I concur this is an overly aggressive rule and have removed it.

Updated version is committed. Closing this thread. :)

[jwpemail]

I strongly object. So far zero evidence has been provided to show legitimate email being sent that is blocked by that rule. We do have evidence, and history, to show that the rule in question blocks SPAM and Viruses. I find it very disheartening that it was so easy to get a well established rule so easily discarded.

[stevejenkins]

At the risk of sounding cheeky, jimpop, this would have been much more helpful feedback to me than your earlier comment of merely "Who sends valid email with a poorly configured rDNS PTR? That's almost insane as few places would easily accept it." I didn't understand what you were trying to communicate.

I hope you'll be less disheartened that I'm happy to put the rule back in, until such time as legitimate evidence can be provided that it's actually blocking legit email.

I appreciate your insight and input -

[quanah]

We have legitimate emails being blocked by it, as I already stated. Since we removed the rule, spamhaus has been blocking those ranges where it's actually invalid, which is preferable in our case. The specific end group that we encounter this with is a security firm (Netragard).

Nov 9 08:31:17 edge02 postfix/smtpd[50403]: NOQUEUE: reject: RCPT from unknown[38.83.192.4]: 554 5.7.1 <4.0-27.192.83.38.in-addr.arpa>: Unverified Client host rejected: Generic - Please relay via ISP; from=xxx@netragard.com to=xxx@zimbra.com proto=ESMTP helo=

[quanah]

in fact, RBLs blocked every in-addr that was invalid before even getting to this rule. So it was only rejecting valid email.

Nov 9 08:10:35 edge01 postfix/postscreen[6524]: NOQUEUE: reject: RCPT from [167.63.39.65]:53116: 550 5.7.1 Service unavailable; client [167.63.39.65] blocked using zen.spamhaus.org; from=romanova18820@mail.ru, to=xxx@zimbra.com, proto=ESMTP, helo=<65.39.63.167.in-addr.arpa> Nov 9 10:30:33 edge02 postfix/postscreen[19838]: NOQUEUE: reject: RCPT from [200.52.84.179]:2106: 550 5.7.1 Service unavailable; client [200.52.84.179] blocked using b.barracudacentral.org; from=keith.adams@oldham.gov.uk, to=xxx@telligent.com, proto=ESMTP, helo=<179.84.52.200.in-addr.arpa>

[quanah]

I'd also note in the case where valid email was being blocked, the helo was correct (mail.netragard.com). markup ate it though. Here it is one again in a code block:

Nov 9 08:31:18 edge02 postfix/smtpd[50403]: NOQUEUE: reject: RCPT from unknown[38.83.192.4]: 554 5.7.1 <4.0-27.192.83.38.in-addr.arpa>: Unverified Client host rejected: Generic - Please relay via ISP; from=<xxx@netragard.com> to=<xxx@zimbra.com> proto=ESMTP helo=<mail.netragard.com>

[jwpemail]

Those may be legitimate emails, but they are being sent from a misconfigured system(s). It's not proper to ask everyone else to lower their shields in order to accommodate someone else's misconfigued system(s).

[plobbes]

Those may be legitimate emails, but they are being sent from a misconfigured system(s).

In a perfect world, maybe. However, the system itself may not be misconfigured at all, but they are certainly using network services from someone who doesn't delegate rDNS to the company to setup as they please. This isn't too unusual in my own experience, in fact I've found it to be all too common. Trying to convince your SP to update rDNS can be difficult or sometimes nearly impossible.

Penalizing a company because they happen to end up w/rDNS ending in in-addr.arpa seems like an overly broad stroke to impose on everyone. I doubt there's any good data to show if a rDNS name ending like this is proven to be more or less spammy than so many other possible names so ultimately it is just a judgement call.

My preference would be that this list not take such a broad stroke in the fight against spam. RBLs help take over for whatever gaps this may leave, but at least those with no rDNS control get a chance while we still get the benefit of all the other rules that remain.

[jwpemail]

So why is the project called fqrdns.pcre, why not just call it misc.pcre?

[moisseev]

As far as I understand fqrdns.pcre is mostly intended to block servers with dynamic and generic rDNS. So /.in-addr.arpa$/ is generic. In my own experience almost every ISP do change rDNS on consumer request. In most cases it is just mail system administrator ignorance. Many of that systems (not all of them) are potential spam and viruses sources due to poor configuration. I'm very sure every generic rDNS should be blocked unless there is a really good reason not to.

As readme says:

If it blocks a sender you know to be legit and from whom you need to receive mail I recommend whitelisting that source instead of removing an expression below as a means to solve the problem.

Why not to create white list for known legitimate servers as part of the fqrdns.pcre project? Presumable it should be a CIDR table.

[stevejenkins]

@moisseev: A whitelist is beyond the scope of this project, but if you or anyone else wants to start one here on Github, I'm more than happy to link to it from the README as as complementary project. :)

[stevejenkins]

Just so I can keep track (and by saying this, I'm not saying it's a democracy... I'm just trying to see where everyone is coming from), @quanah & @plobbes believe this entry should be removed, while @jimpop-at-gmail-dot-com & @moisseev believe it should stay. After discussing and considering, have any of you been swayed the other way?

I'm wondering if it might be useful to also start/move this discussion on the Postfix users list, or perhaps even better, make a post over there inviting others who wish to chime in to do so here.

My general hope for this file is for it to continue in the same spirit as when Stan Hoeppner originally started it, where the rules lean conservative, while more aggressive blocking (or more liberal whitelisting) can be handled individually by the mail admin after-the-fact.

[jwpemail]

My opinion is the same. The rule is necessary, and in-line with the project's goal. Further, it is far easier for 1 or 2 people to remove the rule, rather than requiring 4000 or 5000 other users to add the rule after they download an updated version of fqrdns.pcre.

[stevejenkins]

Received this feedback via email from Noel Jones of the Postfix Users list:

The greatest feature of fqrdns.pcre is its goal of zero false positives. Don't break that. Remember, you're deciding for everyone what what mail they won't receive. Individual users are still free to add their own stricter rules.

Generalized rules that aren't known 100% safe should be avoided. At the time in-addr.arpa PTR was added to the list, there were no known ISPs using those for legit hosts. Now there are, so it's reasonable to remove that entry. As an alternative, a "38.in-addr.arpa DUNNO" whitelist entry might be sufficient for now, but removing the offending rule completely is the safe course. Again, individual users are free to be more strict with their local rules.

[jwpemail]

The "zero false positive" thought process would basically invalidate any and all of the rules. Any and everybody can come up with an excuse to show how a rule causes them some inconvenience.

[stevejenkins]

So I guess the question boils down to: is this rule doing more harm than good?

And what are the chances that a system that gets "caught" by this line wouldn't have been caught by something else (RBL, normally-used Postfix restrictions, etc.)?

The goal of zero false positives will likely never be reached by anything, but it's still the ideal for which to strive.

It's clear we're not going to reach consensus. I'm just looking for more people to chime in. Real world data and examples carry more weight in my mind vs. "what if" scenarios.

[jwpemail]

Just don't forget that DNSBLs are after-the-fact (aka after some spam has started spewing), whereas NXDOMAIN and fqrdns.pcre are before the spew can begin.

[stevejenkins]

I don't want perfection to be the enemy of the good. We'll never achieve perfection, but that doesn't mean we shouldn't try for good.

I think the best approach, which is in-line with the original intent of the project, will always be the most conservative one, then allow individual admins to add their own more restrictive rules, based on their needs.

I haven't made a decision yet, but I will say that I'm inclined to believe the argument that when Stan originally added in-addr.arpa PTR to this list, there were no known ISPs using those for legitimate hosts. I'm also inclined to believe that's no longer the case, which is a believable argument for removal from a list whose primary purpose is supposed to be avoiding false positives, even at the cost of allowing a few more things through to be dealt with by other means in subsequent steps.

If the ./in-addr.arpa$/ is removed from the file, that isn't the same as whitelisting those hosts. That doesn't mean mail from those systems will automatically get through. It simply means that they won't automatically be blocked.

I've asked for real-world examples, not hypothesis. So far, I've only seen log file examples of valid mail that was being blocked by this rule. I haven't seen examples of invalid mail that have been allowed by this rule -- and that wouldn't have also been blocked by standard Postfix best practices.

I'd love to see some examples of that, if any of you have them.

I accept that the original README says:

If it blocks a sender you know to be legit and from whom you need to receive mail I recommend whitelisting that source instead of removing an expression below as a means to solve the problem.

But "a sender you know to be legit and from whom you need to receive mail " is different than "multiple senders that could be legit and from whom you might receive mail," particularly when things have changed since the origin of the file so that in-addr.arpa hosts now might be valid.

So if you have real-world examples of invalid in-addr.arpa mail that wouldn't be stopped by basic Postfix preventative measures, for which the fqrdns.pcre file would have been your only reasonable hope, now is the time to share them. :)

[jwpemail]

This is really getting out of hand. in-addr.arpa is generic, fqrdns.pcre exists to block generic. Competent MTA admins know how to properly accept and deliver email without opening holes in their process.

I've tired of explaining the above, so I'll just avoid fqrdns.pcre updates because apparently it's easier to get a rule removed than it is to explain what a generic addr is.

[stevejenkins]

@jimpop-at-gmail-dot-com I've tried to be polite to you, and listen to all inputs (including yours) before making a decision. You came out of the gates with a rude and dismissive tone, and don't seem to have backed down from it.

"Getting out of hand" seems a tad excessive. This is mail administration, not the battle against ISIS. Stan had the "benefit" of being able to make changes to the file without version history, and would have simply pushed out changes as he saw fit. When I volunteered to take this over, I put it here on GitHub so the process would be more open, and therefore I'm voluntarily open to more scrutiny in my stewardship of the list.

I'm also tired of you explaining the above.. but only because of your insulting tone, and not your expertise -- which I value, as I do that of any others who are generous enough to share it here.

[stevejenkins]

@jimpop-at-gmail-dot-com: Additionally, as I try to look past your complaints to the content of your comments, I note that you didn't provide what I asked for, which is real-world examples of situations where removing this line would be detrimental, rather than esoteric and hypothetical scenarios. I've given you a path to get what you want. I don't understand why you're not taking it.

And if I do end up removing this line, it won't be based on any negative feelings on my part toward you as a result of your rudeness. That would be petty, and that's not my style.

I'm asking for data, so that I can make a data-driven and emotionless decision.

[patrickdk77]

I would think, likely keeping the list as is, and splitting into 3 new ones, this being a combination of the 3, and the 3 other ones based on how likely false positives are to result, would be a good idea, and let people opt themselves how risky to be. As there really is no right choice to be made.

Given the results on my system, I get the following: 269 Blocked via RBL's 1 Blocked due to invalid ptr name (not valid dns name, invalid chars in name, not blocked from this rule, false positive) 6 Blocked due to pipelining 3 Blocked due to invalid from address 1173 Blocked due to this rule, 6 of them cause the dns server is missing the . on the end, but all still spam.

These are the results based on the last year of the company mailserver, and the last 3 months of my personal server.

[stevejenkins]

@patrickdk77 First, thank you for taking the time to dig in your log files to produce some data.

Second, just to confirm, you saw only one false positive from this rule out of 1452 connections? Any rough idea of how many of those 1,173 would have been snagged by other methods?

One false positive out of that many connections really isn't a lot. Very helpful info.

Your idea about splitting the list into a Goldilocks-style approach (Baby Bear, Mama Bear, Papa Bear) with varying levels of risk is an interesting one. What criteria would you propose for determining what types of rules belong on which list? What type of performance penalty would occur if Postfix had to evaluate three PCRE files vs. only one?

[patrickdk77]

No, the 1 false positive was not cause of this rule, but could have
been, It was blocked by other rules, before it hit this list.

It was blocked cause the dns ptr named contained a / char in it.

N/A.xxx.xxx.in-addr.arpa

Quoting Steve Jenkins notifications@github.com:

@patrickdk77 First, thank you for taking the time to dig in your log
files to produce some data.

Second, just to confirm, you saw only one false positive from this
rule out of 1452 connections? Any rough idea of how many of those
1,173 would have been snagged by other methods?

One false positive out of that many connections really isn't a lot.
Very helpful info.

Your idea about splitting the list into a Goldilocks-style approach
(Baby Bear, Mama Bear, Papa Bear) with varying levels of risk is an
interesting one. What criteria would you propose for determining
what types of rules belong on which list? What type of performance
penalty would occur if Postfix had to evaluate three PCRE files vs.
only one?

---

Reply to this email directly or view it on GitHub: https://github.com/stevejenkins/hardwarefreak.com-fqrdns.pcre/issues/4#issuecomment-157063496

[stevejenkins]

Ahhh... ok. So there's no evidence in your log files that you blocked any valid mail as a result of this rule being in the file.

[patrickdk77]

Yes, based on my results, all 1173 where NOT snagged by other rules, as I filtered those out above, 1452 total attempts would match this rule, but only 1173 made it to this rule.

The 1 false positive, would have been snagged by this rule, BUT it also matched other rules before it got this far.

Sorry, just reading this on the website now, the email ticket update I got seemed to truncated the question, and I didn't see all of it.

As far as the issues that could happen with multible lists, I see no issues, yes there will be more effect to process 3 vs 1, but not much, just open/read commands. If people do it like I do atleast, I have an update script that collects the rules, sanity checks them, and compiles them into 1 or a few files, to get published to my servers. So the effect for me, would be 0, as I would be preprocessing them.

The real slowdown is how many regex lines have to be evaluated, I imaging using IF clauses to exclude large sets of regex lines will cause much more benefit. But I have to double check, if this is valid in these maps, and not specific to header checks.

[stevejenkins]

There already are a couple of IF statements in the PCRE file, though they're pretty simple. It first checks to see if certain key words (like dynamic) appear, and if they do, it checks against that subset. If not, it checks against the manually entered hosts in the bottom half of the file.

[stevejenkins]

As an update, I am considering the idea of expanding this project into three separate PCRE files:a primary fqrdns.pcre file (that leans heavily toward avoiding false positives), a fqrdns-plus.pcre file that contains rules that would be deemed "slightly" more aggressive and that have been migrated from the primary file (such as the iana rule under discussion here) that are wanted by some but not by others, and a fqrdns-max.pcre file that contains far more general rules that seek to block the vast majority of dynamic hosts.

To tinker with this idea, I've created a "develop" branch of this project and populated it with a first draft of what such files might look like. Again, this is an experiment, but one in which I welcome polite and thoughtful feedback.

[stevejenkins]

New three-file approach has been running fine for over 18 months, so I've merged it to the master branch.