Posting and configs

[SteffanCline]

I seem to have this working on CentOS 6. I can share my binaries if needed. Some of the configs don't work as expected though. Dovecot does not resolve the $mail_plugins. I had to do it the way I show it below.

protocol imap {
#        mail_plugins = imap_xaps $mail_plugins
    mail_plugins = imap_xaps
}

protocol lda {
#   mail_plugins = xaps $mail_plugins
    mail_plugins = xaps
}

protocol lmtp {
#   mail_plugins = xaps $mail_plugins
        mail_plugins = xaps
}

plugin {
    xaps_socket = /var/run/xapsd/xapsd.sock
}

[SteffanCline]

Ok, thought it was working but apparently not. I never get any push notifications but I don't see any errors in the log. Everything is written to /var/log/xapsd.log. I can send it to you if needed but I can't see where it fails. Don't see any notifications being logged in there. I see other things like

2015/09/29 12:23:47 [DEBUG] Opening database at /var/lib/xapsd/database.json
2015/09/29 12:23:47 [DEBUG] Listening on UNIX socket at /var/run/xapsd/xapsd.sock
2015/09/29 12:23:47 [DEBUG] Parsing /etc/xapsd/certificate.pem to obtain APNS Topic
2015/09/29 12:23:47 [DEBUG] Topic is com.apple.mail.XServer.d19011df-561b-444d-ac59-09459e74a5e6
2015/09/29 12:23:47 [DEBUG] Creating APNS client to gateway.push.apple.com:2195
2015/09/29 12:23:47 Starting xapsd 1.0b1 on /var/run/xapsd/xapsd.sock
...
2015/08/03 06:14:52 [DEBUG] Accepted a connection
2015/08/03 06:14:52 [DEBUG] Received request: REGISTER aps-account-id="E29E6656-76CC-46A8-AD97-A13FF9FC9AEA"    aps-device-token="84219CEACFF782DDEB39911D4CEEC7FD8F3348C3E9605CCB4D4EBE121350EB84" aps-subtopic="com.apple.mobilemail" dovecot-username="chris.buxtonfamily"   dovecot-mailboxes=("INBOX")
2015/08/03 06:14:52 [DEBUG] Returning success: com.apple.mail.XServer.d19011df-561b-444d-ac59-09459e74a5e6
2015/08/03 06:14:54 [DEBUG] Accepted a connection
2015/08/03 06:14:54 [DEBUG] Received request: REGISTER aps-account-id="E29E6656-76CC-46A8-AD97-A13FF9FC9AEA"    aps-device-token="84219CEACFF782DDEB39911D4CEEC7FD8F3348C3E9605CCB4D4EBE121350EB84" aps-subtopic="com.apple.mobilemail" dovecot-username="chris.buxtonfamily"   dovecot-mailboxes=("INBOX","Sent Messages")
2015/08/03 06:14:54 [DEBUG] Returning success: com.apple.mail.XServer.d19011df-561b-444d-ac59-09459e74a5e6

Notice there is no NOTIFY. Which is at fault here?

I sent a mail through and as you can see, nothing was logged here but it was received in the postfix log.

Sep 29 12:34:33 hosting1 postfix/smtpd[2950]: connect from sudnp799.qwest.com[155.70.32.99]
Sep 29 12:34:34 hosting1 postfix/smtpd[2950]: 35EED2F57A3: client=sudnp799.qwest.com[155.70.32.99]
Sep 29 12:34:34 hosting1 postfix/cleanup[2780]: 35EED2F57A3: message-id=<852B66B64526014DB465219F2EC2F805B214A10B@PDDCWMBXEX501.ctl.intranet>
Sep 29 12:34:34 hosting1 postfix/qmgr[741]: 35EED2F57A3: from=<Steffan.Cline@CenturyLink.com>, size=12002, nrcpt=1 (queue active)
Sep 29 12:34:34 hosting1 postfix/smtpd[2950]: disconnect from sudnp799.qwest.com[155.70.32.99]
Sep 29 12:34:35 hosting1 spamd[32682]: spamd: connection from localhost [127.0.0.1] at port 40254
Sep 29 12:34:35 hosting1 spamd[32682]: spamd: setuid to steffan.hldns succeeded
Sep 29 12:34:35 hosting1 spamd[32682]: spamd: processing message <852B66B64526014DB465219F2EC2F805B214A10B@PDDCWMBXEX501.ctl.intranet> for steffan.hldns:527
Sep 29 12:34:43 hosting1 spamd[32682]: spamd: clean message (-1.9/5.0) for steffan.hldns:527 in 8.5 seconds, 11935 bytes.
Sep 29 12:34:43 hosting1 spamd[32682]: spamd: result: . -1 - BAYES_00,HTML_MESSAGE scantime=8.5,size=11935,user=steffan.hldns,uid=527,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=40254,mid=<852B66B64526014DB465219F2EC2F805B214A10B@PDDCWMBXEX501.ctl.intranet>,bayes=0.000000,autolearn=ham
Sep 29 12:34:43 hosting1 postfix/local[2784]: 35EED2F57A3: to=<steffan.hldns@hosting1.hldns.com>, orig_to=<Steffan@hldns.com>, relay=local, delay=9.5, delays=0.24/0/0/9.3, dsn=2.0.0, status=sent (delivered to command: /usr/bin/procmail-wrapper -o -a $DOMAIN -d $LOGNAME)
Sep 29 12:34:43 hosting1 postfix/qmgr[741]: 35EED2F57A3: removed
Sep 29 12:34:43 hosting1 spamd[32660]: prefork: child states: II

It seems the plugin is loading:

# dovecot -a | grep plugin
mail_plugin_dir = /usr/lib64/dovecot
mail_plugins = 
plugin {
  mail_plugins = xaps
  mail_plugins = xaps
  mail_plugins = imap_xaps

in /usr/lib64/dovecot/

# ls -l | grep xaps
-rwxrwxr-x 1 root root  13606 Aug  2 16:17 imap_xaps_plugin.so
-rwxrwxr-x 1 root root  11420 Aug  2 16:17 xaps_plugin.so

The only thing I can think of is that the plugin may have an issue matching up the name that registers with the service versus what's in the log. Is that possible?

Suggestions?

[SteffanCline]

Seems NOTIFY is the cause. Not seeing anything that stands out in the plugin source.

[Decanus5]

I've just spent the day trying to figure this out.

I could see nothing that stood out in the logs, except for the fact I was only ever seeing IMAP in the debug logs, never any LDA or LMTP.

I tried everything eventually I figured out that my Exim transport was set to append to the file and not actually run through Dovecot's LTA/LMTP process. Once I changed this I noticed that it immediately began working.

Hopefully this is the solution you're looking for.

[SteffanCline]

This is a copy and paste from the previous issue which seems related.

I'n in the same boat as @georgeboot I use Virtualmin and when it sets up the server, it configures procmail to do the delivery and dovecot simply does the IMAP transport. I'm wondering if the issue with it not working is because dovecot-lda is not moving the mail so it doesn't know to notify. I too get no notify. You can see my notes on a separate report. I asked @st3fan for help over Twitter and am hoping he may have some insight. I don't know how to approach this to fix it. When I tried putting in the

mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"

mail stopped being delivered.

Any conclusive answer on this?

[Decanus5]

I've actually just finished this. I too had issues with dovecot-lda not delivering. It turned out to be permission based errors.

Thought at the time I posted the message to you I could see in the xaps daemon log the notify by manually running

/usr/libexec/dovecot/dovecot-lda -f "test@test.com" -a "testrecipient@test.com" test⌃d

This generated a notify event and delivered correctly through dovecot-lda, I just couldn't get the transport in Exim to deliver to dovecot-lda.

The errors related to permissions for the socket /var/run/dovecot/auth-userdb

/var/log/maillog helped me identify where I was missing permissions from.

[Decanus5]

The XAPS daemon is only notified on delivery into dovecot by either LDA or LMTP, Your MTA needs to deliver via one of those mechanisms.

Dovecot simply reading the maildir isn't going to generate the necessary event to push the notification to XAPSD. This was the issue I faced earlier today, I was simply appending in Exim rather than pushing to Dovecot.

Does Virtualmin not support a more modern MTA like Postfix, Sendmail or Exim?

[SteffanCline]

It uses postfix as the MTA, procmail for delivery to mailbox and dovecot for IMAP access. I don't know if there's a way to change that. I wonder if there's a way to simply generate a notify without changing the default install.

Seems there is no easy way to change away from procmail https://virtualmin.com/node/19624

[Decanus5]

You could bypass procmail as the delivery mechanism and configure postfix to pass directly to dovecot-lda.

http://wiki.dovecot.org/LDA/Postfix

Regarding modification from default as far as I know there would be no other way without writing a separate process to hook into something like incron/inotify.

Have you tried manually running the dovecot-lda command and seeing if this generates a notify event for you?

[SteffanCline]

I'm reading up on some things right now that suggest you can pipe the output of procmail into dovecot to solve the problem. Yes, using your command with a slight modification, I did get a push notification. That part is excellent! I'll post back what I find.

[SteffanCline]

I got it to work! This is the solution for Virtualmin installs. Mail MUST be routed by dovecot for it to work. http://blog.lorenzbausch.de/2013/07/15/virtualmin-enable-sieve-for-dovecot-and-procmail/

[Decanus5]

Excellent. Glad you got it working.

Big thanks to @st3fan for coding this!

[SteffanCline]

Something else popped up. I posted this to the dovecot list to see but perhaps someone here may know too.

I’m posting this here and will in a couple other places to see if anyone has any experience with this and knows what I can do.

My setup is CentOS 6 with postfix, procmail and dovecot administered via Virtualmin.

Postfix of course receives the email. Procmail uses it’s hooks to run the email through spam assassin and clamd. After than procmail hands it off to dovecot/deliver to drop it in the user’s mail box.

I use the following plugin for dovecot with the daemon to provide Apple push mail for iPhones.

https://github.com/st3fan/dovecot-xaps-daemon

https://github.com/st3fan/dovecot-xaps-plugin

For the most part, this works awesome but have run into a bit of a race issue with notification of mail in a particular instance.

On my computer I use Outlook for Mac which says it supports IDLE but is broken. When an email comes in, I ALWAYS get the notification via the dovecpt plugin for a push since Outlook is checking IMAP every 2 minutes.

On a buddy’s setup, he uses Apple Mail which supports IDLE that actually works. When his email client is connected from his laptop, it does actually maintain a persistent connection and NEVER receives a notification from the dovecot plugin because I’m guessing dovecot/deliver sees that the email was received by the desktop client.

I’m not sure how to make the plugin ALWAYS give a push. Is it a dovecot config? A bug in the plugin? I’m not sure.

Suggestions?

[SteffanCline]

# doveconf -n
# 2.0.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-71.29.1.el6.x86_64 x86_64 CentOS release 6.7 (Final)
auth_mechanisms = plain login
disable_plaintext_auth = no
mail_debug = yes
mail_location = maildir:~/Maildir
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy include variables body enotify environment mailbox date
mbox_write_locks = fcntl
passdb {
  driver = pam
}
plugin {
  sieve = ~/.dovecot.sieve
  sieve_dir = ~/sieve
  xaps_socket = /var/run/xapsd/xapsd.sock
}
postmaster_address = postmaster@hldns.com
protocols = imap pop3 sieve
service managesieve-login {
  inet_listener sieve {
    port = 4190
  }
}
ssl_ca = </etc/dovecot/dovecot.ca.pem
ssl_cert = </etc/pki/dovecot/certs/dovecot.pem
ssl_cipher_list = DEFAULT:!EXPORT:!LOW:!MEDIUM:!MD5
ssl_key = </etc/pki/dovecot/private/dovecot.pem
userdb {
  driver = passwd
}
protocol lmtp {
  mail_plugins = " xaps"
}
protocol lda {
  mail_plugins = " sieve xaps"
}
protocol imap {
  mail_plugins = " imap_xaps"
}
protocol pop3 {
  pop3_uidl_format = %08Xu%08Xv
}

[Decanus5]

@SteffanCline I use AirMail2 and this supports IMAP IDLE, AirMail2 sometimes receives the mail before my phone but XASPD always triggers a push event to my phone and this has been working flawlessly since installing it.

Have you set your XASP Daemon into debug mode and seen if the NOTIFY event is triggered when his mail client is open?

[SteffanCline]

Yes and when it's open, the notify is not sent. I think essentially it's seeing the persistent connection and it assumes the mail is notified because it shows in the client.

[SteffanCline]

@Decanus5 Can you compare your doveconf -n against mine?

[Decanus5]

protocol lda {
  mail_plugins = xaps
}
protocol imap {
 imap_idle_notify_interval = 2 mins
  mail_max_userip_connections = 20
  mail_plugins = imap_xaps
}
protocol lmtp {
  info_log_path = /var/log/dovecot/lmtp.log
  log_path = /var/log/dovecot/lmtp-errors.log
  mail_plugins = xaps $mail_plugins
}

These are the only real differences I see in the config, If your MTA is delivering to Dovecot via LDA it should always fire the notification event.

The LDA Is responsible for delivering the mail, therefor the notify event is triggered before the mail is actually delivered (In my understanding, I might be wrong on this; but that's how I believe it works)

[Decanus5]

Oct  8 22:02:30 mail2 spamd[14820]: spamd: got connection over /var/run/spamassassin/spamd.sock
Oct  8 22:02:30 mail2 spamd[14820]: spamd: checking message <st3fan/dovecot-xaps-plugin/issues/10/146684018@github.com> for spamassassin:500
Oct  8 22:02:35 mail2 spamd[14820]: spamd: clean message (-1.2/5.0) for spamassassin:500 in 4.5 seconds, 3592 bytes.
Oct  8 22:02:35 mail2 spamd[14820]: spamd: result: . -1 - AWL,BAYES_00,DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HTML_IMAGE_ONLY_12,HTML_MESSAGE,URIBL_BLOCKED scantime=4.5,size=3592,user=spamassassin,uid=500,required_score=5.0,rhost=localhost,raddr=127.0.0.1,rport=/var/run/spamassassin/spamd.sock,mid=<st3fan/dovecot-xaps-plugin/issues/10/146684018@github.com>,bayes=0.000067,autolearn=no

Your reply from just before in my /var/log/maillog. I enabled XAPSD Debugging and sent a test message

[root@mail2(/)> tail -f /var/log/xapsd.log
2015/10/08 22:10:45 [DEBUG] Opening database at /var/lib/xapsd/database.json
2015/10/08 22:10:45 [DEBUG] Listening on UNIX socket at /var/run/dovecot/xapsd.sock
2015/10/08 22:10:45 [DEBUG] Parsing /etc/xapsd/certificate.pem to obtain APNS Topic
2015/10/08 22:10:45 [DEBUG] Topic is com.apple.mail.XServer.f1d18f89-6d03-4577-a74d-xxxxxxxxxxx
2015/10/08 22:10:45 [DEBUG] Creating APNS client to gateway.push.apple.com:2195
2015/10/08 22:10:45 Starting xapsd 1.0b1 on /var/run/dovecot/xapsd.sock
2015/10/08 22:11:50 [DEBUG] Accepted a connection
2015/10/08 22:11:50 [DEBUG] Received request: NOTIFY dovecot-username="xxxxxx"  dovecot-mailbox="INBOX"
2015/10/08 22:11:50 [DEBUG] Sending notification to 392F333C-2296-41A1-8C47-xxxxxxxxxx / CAB4D66ED96631A6E50981C633F661A4587CF07C2F2FC61485xxxxxxxxxxx
2015/10/08 22:11:50 [DEBUG] Returning success:

[root@mail2(/)> tail -f /var/log/dovecot/debug.log
Oct 08 22:05:40 lda(xxxxx): Debug: xaps_transaction_begin
Oct 08 22:05:40 lda(xxxxx): Debug: xaps_save_finish
Oct 08 22:05:40 lda(xxxxx): Debug: xaps_transaction_commit
Oct 08 22:05:40 lda(xxxxx): Debug: xaps_plugin_deinit
Oct 08 22:11:49 lda(xxxxx): Debug: xaps_plugin_init
Oct 08 22:11:49 lda(xxxxx): Debug: xaps_transaction_begin
Oct 08 22:11:49 lda(xxxxx): Debug: xaps_transaction_begin
Oct 08 22:11:49 lda(xxxxx): Debug: xaps_save_finish
Oct 08 22:11:50 lda(xxxxx): Debug: xaps_transaction_commit
Oct 08 22:11:50 lda(xxxxx): Debug: xaps_plugin_deinit

[SteffanCline]

@Decanus5 I did get it to work by modifying the delivery. procmail gets the mail from postfix, scans it through clamd and spam assassin then procmail gives it to deliver. So, deliver/dovecot is getting the hand off but seems to be some kind of config or plugin issue.

[SteffanCline]

# cat /etc/procmailrc
DELIVER=/usr/libexec/dovecot/deliver
LOGFILE=/var/log/procmail.log
TRAP=/etc/webmin/virtual-server/procmail-logger.pl
:0wi
VIRTUALMIN=|/etc/webmin/virtual-server/lookup-domain.pl --exitcode 73 $LOGNAME
EXITCODE=$?
:0
* ?/usr/bin/test "$EXITCODE" = "73"
/dev/null
EXITCODE=0
:0
* ?/usr/bin/test "$VIRTUALMIN" != ""
{
INCLUDERC=/etc/webmin/virtual-server/procmail/$VIRTUALMIN
}
DEFAULT=$HOME/Maildir/
ORGMAIL=$HOME/Maildir/
DROPPRIVS=yes
:0 w
| $DELIVER
:0
$DEFAULT

[Decanus5]

Try manually running /usr/libexec/dovecot/deliver destination@domain.com type a message then press ctrl+d to end input and deliver, this should give a notification on both the IMAP IDLE client and the Apple device. If this is happening then we've narrowed it down to the procmail segment.

[SteffanCline]

That does work like it did before. Something quirky is going on. I'm going to use the logging options you did to dump the plugin debug info into the xapsd log since it's related and make it easier to follow. (xapsd, dovecot, postfix maillog)

[freswa]

Since the backend changed, I doubt this issue still exist. If anyone is affected by this again please reopen. Thanks