search

virtualmin / virtualmin-gpl

Virtualmin web hosting control panel for Webmin
https://www.virtualmin.com
GNU General Public License v3.0
315 stars 97 forks source link

Feature: milter-greylist: GeoIP is not available - Has been broken from AFAIK for years.. #182

Closed abclution closed 3 years ago

abclution commented 4 years ago

Legacy GeoIP database updates are unavailable in the format needed.

Additionally enabling the distro distributed database in /etc/milter-greylist/greylist.conf via geoipdb "/usr/share/GeoIP/GeoIP.dat" breaks completely the milter-greylist service. (Debian 10)

This is a terribly old issue, more info:

In r490200, GeoIP support was authoritatively removed. The reason was that Maxmind no longer provides geolocation data in the legacy format used by net/GeoIP, which milter-greylist uses. (https://svnweb.freebsd.org/ports?view=revision&revision=490211). While there is a UPDATING record, this action has an unexpected and big impact for users already using GeoIP databases with milter-greylist. It had to be left to their decision if they want to use outdated data, or abandon GeoIP support at all. Especially in the milter-greylist case, I can't imagine anybody would consider removing GeoIP support was the correct solution. I guess they would live with outdated data until somebody skilled spends a significant amount of time to implement the successor library net/libmaxminddb, which we already have in the ports tree.

Solution is a fork of milter-greylist that supports the new maxmind database format:

Thankfully, Markus Wennrich did that job. His decision was to publish a complete fork on github: https://github.com/mwennrich/milter-greylist-geoip2

More info here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239334

Suggestion: I don't know, not really fun to package new packages with Virtualmin, perhaps removing milter-greylist support is an easier option. Just wanted to bring it to attention.

jcameron commented 4 years ago

Virtualmin just uses the milter-greylist package supplied by the underlying OS vendor (Debian, Ubuntu, CentOS, etc). So any fix to use a new source of GeoIP data would have to be made there..

swelljoe commented 4 years ago

Why does milter-greylist even use GeoIP? I don't think we care about that information at that point in the process...I certainly didn't know it had that feature.

Regardless, I definitely don't want to maintain another package for a rarely used (maybe even useless, given that SpamAssassin already has it?) feature. Do we enable GeoIP in milter-greylist by default? Or does the package do so on Debian? Seems like a bad decision if so. Maybe we should just disable it by default. Folks who want to do something with GeoIP in the greylist can use the fork...but, maybe best to just not?

abclution commented 4 years ago

@swelljoe Pretty much agree with that.

milter-greylog is very noisy about the issue in mail.log with no known way to repress it complaining every 1-2 seconds.

Jul 11 14:52:00 vps1 milter-greylist: GeoIP is not available And it silently fails completely if anyone follows the old instructions to enable it.

I wonder what greylisting package Proxmox Mail Gateway uses, did they roll their own .... hmmm

I guess the other option is to poke the bears upstream about repackaging the fixed version as well.

swelljoe commented 4 years ago

What old instructions to enable it are you talking about? Is it something we've published? If so, I can fix it. I don't see any reason for GeoIP to be enabled (and I don't see any warnings like this on our systems with greylisting enabled).

abclution commented 4 years ago

@swelljoe

Old instructions / solutions being those found floating on the net when encountering "GeoIP is not available" this in the /var/log/mail.log and searching the great Googly Moogly for solutions. This includes old virtualmin forum posting and other random places. Not any specific instructions..

19 seconds of tailing /var/log/mail.log give me 11 instances of "Jul 12 12:30:20 vps1 milter-greylist: GeoIP is not available" (Debian 10)

Jul 12 12:30:20 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:22 vps1 postfix/smtpd[26657]: connect from unknown[46.38.145.247]
Jul 12 12:30:22 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:22 vps1 postfix/smtpd[30919]: warning: unknown[46.38.150.190]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:22 vps1 postfix/smtpd[30919]: disconnect from unknown[46.38.150.190] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:23 vps1 postfix/smtpd[14830]: connect from unknown[185.143.73.162]
Jul 12 12:30:23 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:25 vps1 postfix/smtpd[14830]: warning: unknown[185.143.73.162]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:25 vps1 postfix/smtpd[23879]: warning: unknown[46.38.150.142]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:25 vps1 postfix/smtpd[23879]: disconnect from unknown[46.38.150.142] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:26 vps1 postfix/smtpd[14830]: disconnect from unknown[185.143.73.162] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:28 vps1 postfix/smtpd[29695]: warning: unknown[212.70.149.82]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:28 vps1 postfix/smtpd[3190]: warning: unknown[212.70.149.19]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:28 vps1 postfix/smtpd[26657]: warning: unknown[46.38.145.247]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:29 vps1 postfix/smtpd[26657]: disconnect from unknown[46.38.145.247] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:29 vps1 postfix/smtpd[3190]: disconnect from unknown[212.70.149.19] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:29 vps1 postfix/smtpd[29695]: disconnect from unknown[212.70.149.82] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:30 vps1 postfix/smtpd[3192]: connect from unknown[46.38.148.22]
Jul 12 12:30:30 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:30 vps1 postfix/smtpd[3196]: connect from unknown[212.70.149.3]
Jul 12 12:30:30 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:31 vps1 postfix/smtpd[30919]: connect from unknown[46.38.150.153]
Jul 12 12:30:31 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:31 vps1 postfix/smtpd[23879]: connect from unknown[46.38.145.251]
Jul 12 12:30:31 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:31 vps1 postfix/smtpd[14830]: connect from unknown[185.143.73.148]
Jul 12 12:30:31 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:36 vps1 postfix/smtpd[3192]: warning: unknown[46.38.148.22]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:36 vps1 postfix/smtpd[26657]: connect from unknown[46.38.145.250]
Jul 12 12:30:36 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:37 vps1 postfix/smtpd[3192]: disconnect from unknown[46.38.148.22] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:37 vps1 postfix/smtpd[3190]: connect from unknown[46.38.148.10]
Jul 12 12:30:37 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:37 vps1 postfix/smtpd[30919]: warning: unknown[46.38.150.153]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:37 vps1 postfix/smtpd[14830]: warning: unknown[185.143.73.148]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:37 vps1 postfix/smtpd[14830]: disconnect from unknown[185.143.73.148] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:37 vps1 postfix/smtpd[30919]: disconnect from unknown[46.38.150.153] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
Jul 12 12:30:38 vps1 postfix/smtpd[29695]: connect from unknown[185.143.72.23]
Jul 12 12:30:38 vps1 milter-greylist: GeoIP is not available
Jul 12 12:30:38 vps1 postfix/smtpd[3196]: warning: unknown[212.70.149.3]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:38 vps1 postfix/smtpd[23879]: warning: unknown[46.38.145.251]: SASL LOGIN authentication failed: authentication failure
Jul 12 12:30:39 vps1 postfix/smtpd[3196]: disconnect from unknown[212.70.149.3] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4
chris001 commented 4 years ago

Ideally, we ask the milter-greylist maintainers to update their code to auto-detect both the old format GeoIP databases, and the new-format GeoIP databases, and read each of them with the proper code so that both work no matter whichever one GeoIP version databases installed.

swelljoe commented 4 years ago

I don't think it's worth spending any effort on. Just don't use that feature. Or, if it matters, use the fork, but I don't see how it can matter that much.

chris001 commented 4 years ago

The value of having GeoIP working at milter-greylist level is so that you can let your MTA (postfix) assign an increased penalty in terms of delay and force more retries, on historically spammy/bad reputation Geographical IP addresses.

So your postfix MTA can make the email scammers from a given GeoIP location, who have historically sent for example 100x more email scams than the average, your MTA will make their MTA wait longer (and experience higher system load), and perform more retries (do more work), than good reputation GeoIP addresses.

GeoIP on milter-greylest results in strongly discouraging spammers/scammers.

chris001 commented 4 years ago

Great news. The fork of milter-greylist which is compatible with the new format GeoIP2/GeoLite2 databases, by @mwennrich, has now been merged upstream to the main repository for milter-greylist, and the new package version 4.6.3 is now released. Watch for the new milter-greylist package to become available on your server, and install it when it becomes available for your Linux distro!

eugenefvdm commented 3 years ago

Sorry any news on this? I want to rid my log files of GeoIP database problems. I have a free GeoLite2 database key.

swelljoe commented 3 years ago

Why would there be news? I explained our position on it above.

eugenefvdm commented 3 years ago

@swelljoe sorry I didn't mean to offend. I was referring to @chris001 who seems to implicate that there is a new version that could possibly resolved the GeoIP issue. Thanks.

chris001 commented 3 years ago

@eugenevdm Which OS distro and version are you running?

Ubuntu Focal 20.04 has the fixed milter-greylist version 4.6.2 https://packages.ubuntu.com/source/focal/milter-greylist

Debian 11 bullseye (testing) and sid (unstable) has milter-greylist version 4.6.2 https://packages.debian.org/search?keywords=milter-greylist

RedHat, Scientific, CentOS 7 and 8 has milter-greylist 4.6.2 https://centos.pkgs.org/7/epel-x86_64/milter-greylist-4.6.2-2.el7.x86_64.rpm.html

Fedora 32, 33, 34, EPEL 7, and EPEL 8, has it. https://src.fedoraproject.org/rpms/milter-greylist

ArchLinux has it. https://aur.archlinux.org/packages/milter-greylist/

FreeBSD 11, 12, and 13 has it! https://www.freshports.org/mail/milter-greylist/

chris001 commented 2 years ago

Ubuntu 22.04 LTS Jammy has the new milter-greylist 4.6.4-1 which runs with no error logs using the new free GeoIP2 database since 4.6.3. @eugenevdm @abclution