Explain how sieve can work instead of maildrop

[Infern1]

Current situation

Maildrop moves messages coming from foo@mail.com to the folder Foo

Desired situation

Migrate those rules to Sieve, so they can be easily managed in roundcube

How can the same achieved by Sieve from Dovecot?

I understand incoming mail flow as: Haraka -> qmail-smtpd -> Maildrop -> vpopmail deliver to Maildir => user reads mail with IMAP

I don't see where Sieve with lmtp replaces maildrop

[msimerson]

I understand incoming mail flow as: Haraka -> qmail-smtpd -> vdelivermail -> maildrop -> deliver to Maildir => user reads mail with IMAP

In 2017 I updated the qmail-deliverabled plugin and Haraka to support this delivery path:

Haraka -> dovecot (LMTP) -> Maildir

It works for standard email accounts and it knows the difference by using the qmail-deliverabled plugin. In that scenario, dovecot is the LDA and sieve scripts can act on messages the same as maildrop in the former. I've been meaning to also update the Haraka aliases plugin so that aliases and forwards also get handled in Haraka.

[Infern1]

So the answer is, if I want to use LMTP with Haraka I should manually change qmail-deliverable.ini the default provision script won't do this for me.

Any example would be good, with some pros and cons, so people can decide

[msimerson]

Correct, the underlying assumption of the provision scripts is that your configs are holy and sacred and should not be trampled upon. Here's how I tell Haraka to deliver via LMTP:

# less qmail-deliverable.ini 
check_outbound=true
host=172.16.15.8
queue=smtp_forward
next_hop=lmtp://172.16.15.15

The key is the next_hop setting. QMD returns different codes for Maildir, email lists, etc. Based on that, if the email address is a Maildir, then Haraka delivers it directly to Dovecot LDA.

Both roundcube and snappymail have plugins for editing sieve scripts. You do have to set up sieve though in dovecot. Here are some relevant bits from my dovecot local.conf file:

protocols = imap pop3 lmtp sieve submission

protocol lmtp {
  mail_fsync = optimized
  mail_plugins = $mail_plugins sieve
}

service lmtp {
  user = vpopmail
  inet_listener lmtp {
    port = 24
  }

  unix_listener lmtp {
    #mode = 0666
  }
}

service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}

[Infern1]

Correct, the underlying assumption of the provision scripts is that your configs are holy and sacred and should not be trampled upon.

That's what I like about the provision script, however sometimes with upgrades I might miss new functionality. I don't mind editing by hand, however it would if people are aware of this new optional stuff.

Here's how I tell Haraka to deliver via LMTP:

# less qmail-deliverable.ini 
check_outbound=true
host=172.16.15.8
queue=smtp_forward
next_hop=lmtp://172.16.15.15

The key is the next_hop setting. QMD returns different codes for Maildir, email lists, etc. Based on that, if the email address is a Maildir, then Haraka delivers it directly to Dovecot LDA.

That is straightforward, any reason not to make it default?

Both roundcube and snappymail have plugins for editing sieve scripts. You do have to set up sieve though in dovecot. Here are some relevant bits from my dovecot local.conf file:

I think this works out of the box

[msimerson]

That is straightforward, any reason not to make it default?

Probably not. I like giving changes some time to "burn in" and surface edge cases. I see that I enabled that a few years ago and haven't had any issues.

I think this works out of the box

Yep, it does appear I added that quite a long time ago as well.

[Infern1]

Yes, this change makes Sieve Filter work!

So default /usr/local/etc/mail/mailfilter is not used when using lmtp ? (Which is not really a problem for me)

.qmail forward are processed as normal I assume

[Infern1]

Did you test Sieve filter with forward to a different address?

2023-05-25T14:54:31.723Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] sender <rob.lensen@foobar.com> code=CONT msg=""
2023-05-25T14:54:31.729Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] recipient <somefamilly@yahoo.com> code=OK msg="not local" sender=rob.lensen@foobar.com
2023-05-25T14:54:31.731Z [INFO] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] client half closed connection ip=172.16.15.15
2023-05-25T14:54:31.733Z [NOTICE] [2DF21179-C2E3-468B-BB41-C80685909F39.1] [core] disconnect ip=172.16.15.15 rdns=dovecot helo=dovecot relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=0/0/1 msgs=0/0/0 bytes=0 lr="550 I cannot deliver mail for <somefamilly@yahoo.com>" time=0.062
2023-05-25T14:54:31.736Z [NOTICE] [CD6D503D-8C40-48DA-9813-28CBA06A70C6.1.1] [outbound] recipient <simone@bla.nu> deferred: 451 4.2.0 <simone@bla.nu> Execution of Sieve filters was aborted due to temporary failure
2023-05-25T14:54:31.737Z [INFO] [CD6D503D-8C40-48DA-9813-28CBA06A70C6.1.1] [outbound] Temp failing 1685026471525_1685026471525_0_91990_IgoAyW_1_haraka for 64 seconds: Some recipients temp failed: <simone@bla.nu>

Fixed by adding to local.conf

# less local.conf |grep sub
submission_host = haraka:25

[msimerson]

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

[Infern1]

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

I think it is exactly the same functionality, however not coming from the provision scripts

[Infern1]

Hmm, I have a similar addition. I should look into what the difference is.

/data/dovecot/etc # grep sub local.conf 
protocols = imap pop3 lmtp sieve submission
service submission {
submission_relay_host = haraka
submission_relay_ssl = no

can you post whole your local.conf cannot get forwards to work correctly or does haraka also need a change?

sieve: info: started log at 2023-06-23 14:16:52 +0200.
error: msgid=<CAEgN0qL0MaS1WY_LeNQuktkyK_LaZiLjSmjZbMeKvnhuX0JS3A@mail.gmail.com>: redirect action: failed to redirect message to <r .XX@xx.com>: Sendmail program returned error (temporary failure).

2023-06-23T12:16:52.209Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound]  hook=get_mx plugin=qmail-deliverable function=hook_get_mx params=bsdfreaks.nl retval=OK msg="[object Object]"
2023-06-23T12:16:52.233Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] secured verified=false cipher=TLS_AES_256_GCM_SHA384 version=TLSv1.3 error=ERR_TLS_CERT_ALTNAME_INVALID cn=bsdfreaks.nl organization="" issuer="Let's Encrypt" expires="Aug  9 21:53:51 2023 GMT" fingerprint=AE:09:22:FA:E2:3E:AB:69:17:D6:9E:4D:E0:85:13:1B:07:07:BC:72
2023-06-23T12:16:52.339Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89] [core] connect ip=172.16.15.15 port=30969 local_ip=172.16.15.9 local_port=25
2023-06-23T12:16:52.345Z [INFO] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89] [helo.checks] skip:proto_mismatch(private)
2023-06-23T12:16:52.424Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] sender <r.j.xx@gmail.com> code=CONT msg=""
2023-06-23T12:16:52.429Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] recipient <r.XX@xx.com> code=OK msg="not local" sender=r.j.xx@gmail.com
2023-06-23T12:16:52.431Z [INFO] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] client half closed connection ip=172.16.15.15
2023-06-23T12:16:52.431Z [NOTICE] [2AB0B96F-C1BF-4A52-B360-8F5FD0D53E89.1] [core] disconnect ip=172.16.15.15 rdns=dovecot helo=dovecot relay=N early=N esmtp=N tls=N pipe=N errors=0 txns=1 rcpts=0/0/1 msgs=0/0/0 bytes=0 lr="550 I cannot deliver mail for <rob.lensen@linde.com>" time=0.094
2023-06-23T12:16:52.434Z [NOTICE] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] recipient <rob@bsdfreaks.nl> deferred: 451 4.2.0 <rob@bsdfreaks.nl> Execution of Sieve filters was aborted due to temporary failure
2023-06-23T12:16:52.434Z [INFO] [BF87DC59-22F7-472F-A079-466AD19EE03C.1.1] [outbound] Temp failing 1687522547321_1687522611581_1_2723_bi9p7d_204_haraka for 128 seconds: Some recipients temp failed: <rob@bsdfreaks.nl>

[msimerson]

Forwards won't yet work via LMTP. QMD only designates vpopmail directories as LMTP eligible, everything else is delivered in the usual way. I did recently fix a couple bugs related to this in Haraka and smtp_forward plugins.

https://gist.github.com/msimerson/e0b5f92582c1a9046ef5fdf27ff200c8

[msimerson]

I've been poking at this some more.

When Haraka delivers to a dovecot Maildir via LMTP, it bypasses qmail, maildrop, and vpopmail. This is desirable if you're moving to sieve as it completely bypasses maildrop and issue #534. It may be surprising to learn that email forwards and vacations defined in qmailadmin no longer work. The mechanism on the backend that implements that is qmail-local delivery rules. If a user has logged into qmailadmin and set up a forward, that forward works by adding an entry such as &user@example.com to the .qmail in their virtual mail directory. If the mailbox is forwarded with 'Save a Copy' checked, then the .qmail file will have two lines. With LMTP delivery to dovecot, both are ignored.

Dovecot LDA can replicate the functionality we've been using maildrop for. The quota plugin replicates the deliverquota program from maildrop. Sieve can implement mail filtering, forwards, and vacation autoreplies.