Upstream from my fork - Githubissues

griff commented 6 years ago

I have created a mail setup for myself that started as a fork of this project but has diverged a great deal since then.

Some of the things I have implemented are:

Switched away from rmilter like #25 suggests
Awstats site for mail generated hourly like #35 suggests
Training of rspamd using imap_sieve. So moving to Junk trains as spam. Moving out of Junk trains ham
Uses OpenDKIM and pypolicyd-spf for DKIM signing and SPF checking
I am working on OpenDMARC support

And now I was wondering if you are interested in these changes?

They come with some caveats and issues.

Rspamd

A newer version of rspamd is needed to get rid of rmilter which is only in unstable so I override it in my config.

I needed more control over the rspamd service configuration so I have forked that into a rspamd2 module that should probably be upstreamed to nixpkgs. It supports socket activation and has much more control over worker configuration.

Awstats

Awstats package does not include tools dir so I had to override the package
The Awstats service in NixOS is geared towards apache stats so I made my own module that generates mail stats and a static site for it. That module doesn't have any security so everyone on the internet can see it. And it only works for one domain.

OpenDMARC

OpenDMARC is not in nixpkgs so I made my own package for it which should probably be upstreamed to nixpkgs.

For OpenDMARC to work it needs a mysql database. I tried to incorporate an sqlite patch someone had done but it was old and outdated.

r-raymond commented 6 years ago

Awesome. I think we should absolutely make use of the work you have invested!

About the Rspamd, I updated the version on nixpkgs and am waiting right now to become stable (which should happen 18.03 I guess). Maybe we can use the window to try to merge your additional config?

The same goes for awstats, I think we could really profit from getting that upstream.

In general I think it would not be wise for this project to have any downstream packages unless there is absolutely no other way.

The junk training might be a little overkill for the goals of the project, but if you already have it working, I'd like to get that in too.

About the OpenDMARC, I don't know if the benefits justify additional dns records that need to be set. I guess some kind of vote on that would be great.

Do you need help to try to get your packages upstream?

Please let this issue open so we can track the progress of the inclusion!

Oh and thanks a lot for all the work!

griff commented 6 years ago

DMARC is a two part process. It is others sending you reports based on DNS records and you sending reports to others based on their DNS records. OpenDMARC only does the latter and you can have one part without the other. So OpenDMARC only makes you a better email citizen but it does require a MySQL DB to keep track of the stats it sends in aggregate reports which kind of sucks.

phdoerfler commented 6 years ago

Training of rspamd using imap_sieve. So moving to Junk trains as spam. Moving out of Junk trains ham

Yes please!

phdoerfler commented 6 years ago

@griff What's the progress on the port? I am very much looking forward to this (especially imap_sieve and your buffed rspamd module).

Personal opinion: I have a long standing personal and subjective feud with MySQL (or MariaDB or how it calls itself) and thus see anything that requires MySQL as inherently flawed. I appreciate your efforts to try make this work with sqlite. Ignoring this very biased view as long as one can turn it off, it's water under the bridge. Fortunately not even GMail requires people to use DKIM (let alone DMARC) for personal mail servers. Of course this is a different case for say sending out newsletters.

griff commented 6 years ago

@phdoerfler right now I am working on upstreaming my changes to hopefully get them included in the upcoming NixOS 18.03. I have completed awstats package (NixOS/nixpkgs#34397) and rspamd module (NixOS/nixpkgs#34562) and am considering what to do about my mail-awstats module and unfinished OpenDMARC module. The spam training code also has a problem in that it works but because dovecot wants to recompile the sieve scripts on each invocation it reports an error in the log each time.

phdoerfler commented 6 years ago

@griff I see your rspamd module has been merged, yay! I'm very excited for 18.03 as this will finally allow to set the password for the rspamd controller (on which https://github.com/r-raymond/nixos-mailserver/issues/16 is depending).

This being said, would you happen to know how this (The 17.09 rspamd module):

      worker {
      ${mkBindSockets cfg.bindSocket}
        .include "$CONFDIR/worker-normal.inc"
      }
      worker {
      ${mkBindSockets cfg.bindUISocket}
        .include "$CONFDIR/worker-controller.inc"
      }

can generate this? (the rspamd.conf file as it is being used by this mailserver project):

worker {
    bind_socket = "*:11333";
    .include "$CONFDIR/worker-normal.inc"
    .include(try=true; priority=1,duplicate=merge) "$LOCAL_CONFDIR/local.d/worker-normal.inc"
    .include(try=true; priority=10) "$LOCAL_CONFDIR/override.d/worker-normal.inc"
}

worker {
    bind_socket = "localhost:11334";
    .include "$CONFDIR/worker-controller.inc"
    .include(try=true; priority=1,duplicate=merge) "$LOCAL_CONFDIR/local.d/worker-controller.inc"
    .include(try=true; priority=10) "$LOCAL_CONFDIR/override.d/worker-controller.inc"
}

Where do these 4 lines with try=true etc. come from? And what file generates rspamd.conf if not this very module?

griff commented 6 years ago

@phdoerfler it looks like the default rspamd.conf that comes with rspamd not the generated one from the module. The module defaults to binding the normal socket to /run/rspamd/rspamd.sock not *:11333

r-raymond commented 6 years ago

any updates on this project? 18.03 is coming, let me know if I can help.

griff commented 6 years ago

I think I might miss the freeze for 18.03 with NixOS/nixpkgs#35536 since it has gotten a little lost in the PR list.

But other than that my plan was to slowly start with what I can do before 18.03. So switching to using OpenDKIM directly and checking SPF with pypolicyd-spf. But to ensure that those two work correctly I was thinking of making a new test with a DNS server and two mail servers so that I can verify DKIM and SPF checking by sending mails between the two servers.

r-raymond commented 6 years ago

Hi, any updates? (Sorry for spamming this issue)

griff commented 6 years ago

@r-raymond Sorry for the delay. Had to go to a wedding/extended family visit on the other side of the globe and that took 3 weeks with 1 week getting over jetlag after getting home.

I am mostly done with OpenDKIM, pypolicyd-spf and moving away from rmilter. I have though one question to you: While working on this I added a test for ClamAV scanning which stores the virus database locally because tests don't have network access. But those files are 157MB so I am a bit reluctant to add them to git and I can't download them from ClamAV with fetchurl since they change daily. Should I just add them to git or do you have some other place they can be stored?

griff commented 6 years ago

@r-raymond I tried using git LFS to store the ClamAV database as described here but it seems that because I don't have permission to upload LFS files to your official repo I can't upload LFS files to my fork.

Other than that I am done with OpenDKIM, pypolicyd-spf and using rpamd proxy worker instead of rmilter.

r-raymond commented 6 years ago

@griff Thanks alot, don't worry, life sometimes is more important! How can I assist with the LFS upload?

r-raymond / nixos-mailserver

Upstream from my fork #61

Rspamd

Awstats

OpenDMARC