Not handling reject response correctly when multiple recipients

[popindavibe]

Similar issues found

3616

Root Cause

My mail server is enforcing a 'reject' rule on unknown recipient domain. So any email containing unknown domain will not be accepted. It works with other MUA I tested, and indeed the 450 error is raised and no mail is sent until the list of recipients is sanitized. In the case of K9-Mail, I end up with a mangled header missing multiple fields (From / Date) and with an 'unexpected end of header'. For Googlemail, the server respond waving RFC 5322 and refuse to accept the email. Others will discard and send a report to the recipient regarding the bad formatted email, and Outlook/Microsoft will accept it but it will be blank for the recipient. On K9-Mail side, the email will be stuck forever in the Outbox, trying to resend the botched email every now and then, never succeeding (as expected).

Expected behavior

Whenever a recipient is rejected, the MUA should handle it gracefully. Testing with Thunderbird and SOGo, they both raise the error and do not try further attempts to deliver the email until the 'bad' recipients have been removed from the list.

K9-Mail should do the same and stop all attempt on receiving the reject from the mail server until the list of recipients has been sanitized.

Actual behavior

Mail server rejects the email, with and from=<> as per bouncing standards, however K9-Mail basically integrate that bad-formatted header and use it to try sending the email to the rest of the recipients.

Steps to reproduce

0. Enforce reject rule on unknown recipient domains on your mail server (before any permit rule)
1. Prepare an email to multiple recipients with at least one 'bad' recipient (ex: unknown domain. Ex: nobody@gmail72.com)
2. Send it from K9-Mail
3. Check your Outbox in K9-Mail and your logs on the mail server

Environment

K-9 Mail version: 5.500

Android version: 8.1.0

Account type (IMAP, POP3, WebDAV/Exchange): IMAP

Please take some time to retrieve logs and attach them here:

Logs from the mail server:

Jul 23 09:44:35 myhost postfix/smtpd[26369]: connect from XXX.XXX.XXX.137.threembb.co.uk[XXX.XXX.XXX.137]
Jul 23 09:44:35 myhost postfix/smtpd[26369]: Anonymous TLS connection established from XXX.XXX.XXX.137.threembb.co.uk[XXX.XXX.XXX.137]: TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)
Jul 23 09:44:35 myhost postfix/smtpd[26369]: CEC231003: client=XXX.XXX.XXX.137.threembb.co.uk[XXX.XXX.XXX.137], sasl_method=PLAIN, sasl_username=obsucated-p@domain.fr
Jul 23 09:44:35 myhost postfix/smtpd[26369]: CEC231003: reject: RCPT from XXX.XXX.XXX.137.threembb.co.uk[XXX.XXX.XXX.137]: 450 4.1.2 <obsucated-t@domain.com>: <b>Recipient address rejected: Domain not found</b>; from=<obsucated-p@domain.fr> to=<obsucated-t@domain.com> proto=ESMTP helo=<[XXX.XXX.XXX.119]>
Jul 23 09:44:36 myhost postfix/cleanup[26374]: CEC231003: message-id=<>
Jul 23 09:44:36 myhost opendkim[32642]: <b>CEC231003: can't determine message sender; accepting</b>
Jul 23 09:44:36 myhost opendmarc[619]: CEC231003: RFC5322 requirement error: missing From field; accepting
Jul 23 09:44:36 myhost postfix/qmgr[26269]: CEC231003: from=<obsucated-p@domain.fr>, size=213, nrcpt=2 (queue active)
Jul 23 09:44:36 myhost postfix/smtp[26375]: Untrusted TLS connection established to gmail-smtp-in.l.google.com[2a00:1450:400c:c0a::1b]:25: TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)
Jul 23 09:44:36 myhost postfix/smtpd[26369]: disconnect from XXX.XXX.XXX.137.threembb.co.uk[XXX.XXX.XXX.137] ehlo=2 starttls=1 auth=1 mail=1 rcpt=2/3 data=1 quit=1 commands=9/10
Jul 23 09:44:36 myhost postfix/smtp[26375]: CEC231003: to=<obsucated-j@domain.com>, relay=gmail-smtp-in.l.google.com[2a00:1450:400c:c0a::1b]:25, delay=0.99, delays=0.4/0.01/0.16/0.43, dsn=5.7.1, status=bounced (host gmail-smtp-in.l.google.com[2a00:1450:400c:c0a::1b] said: 550-5.7.1 [2a01:4f8:10a:2493:b827:a8ff:fe4b:b71c      11] Our system has 550-5.7.1 detected that this message is not RFC 5322 compliant: 550-5.7.1 'From' header is missing. 550-5.7.1 To reduce the amount of spam sent to Gmail, this message has been 550-5.7.1 blocked. Please visit 550-5.7.1  https://support.google.com/mail/?p=RfcMessageNonCompliant 550 5.7.1 and review RFC 5322 specifications for more information. y4si40902182wrr.356 - gsmtp (in reply to end of DATA command))

[cketti]

Can you please record a debug log while trying to send a message. See https://github.com/k9mail/k-9/wiki/LoggingErrors

Also make sure you're not actually using K-9 Mail 5.500. Because that was a beta version. The current stable version is 5.600.

[popindavibe]

hmm, I am using /e/ Android Google-free OS, and though I checked for updates, it appears the App version is currently 5.5000. I opened a thread on their forum, but since I use /e/ as my main and only phone, I will wait until the app, which is called Mail on /e/ and which is based on K9-Mail, gets updated.

If you can keep this open, I'll redo testing and will upload debug log if pb is still there once I get the upgrade.

[cketti]

Please only open bugs that you can actually reproduce using K-9 Mail.

[popindavibe]

Alright, I borrowed a phone, installed K9 5600 from F-Droid, reverted my changes on the mail server, and could reproduce the bug.

Here is the bug report

Also, here is the notification for the sender:

and for the existing recipient, a blank email:

[cketti]

From the log:

D/SmtpTransport(30906): SMTP <<< 220 mail.nomagic.uk ESMTP Postfix (Debian/GNU)
D/SmtpTransport(30906): SMTP >>> EHLO [192.168.20.11]
D/SmtpTransport(30906): SMTP <<< 250-mail.nomagic.uk
D/SmtpTransport(30906): SMTP <<< 250-PIPELINING
D/SmtpTransport(30906): SMTP <<< 250-SIZE 10240000
D/SmtpTransport(30906): SMTP <<< 250-VRFY
D/SmtpTransport(30906): SMTP <<< 250-ETRN
D/SmtpTransport(30906): SMTP <<< 250-AUTH PLAIN LOGIN
D/SmtpTransport(30906): SMTP <<< 250-AUTH=PLAIN LOGIN
D/SmtpTransport(30906): SMTP <<< 250-ENHANCEDSTATUSCODES
D/SmtpTransport(30906): SMTP <<< 250-8BITMIME
D/SmtpTransport(30906): SMTP <<< 250-DSN
D/SmtpTransport(30906): SMTP <<< 250 SMTPUTF8
D/SmtpTransport(30906): SMTP >>> *sensitive*
D/SmtpTransport(30906): SMTP <<< 235 2.7.0 Authentication successful
D/SmtpTransport(30906): SMTP >>> MAIL FROM:<usera@nomagic.uk> BODY=8BITMIME
D/SmtpTransport(30906): SMTP >>> RCPT TO:<userb@nomagic.fr>
D/SmtpTransport(30906): SMTP >>> RCPT TO:<testi@example.com>
D/SmtpTransport(30906): SMTP >>> DATA
D/SmtpTransport(30906): SMTP <<< 250 2.1.0 Ok
D/SmtpTransport(30906): SMTP <<< 250 2.1.5 Ok
D/SmtpTransport(30906): SMTP <<< 556 5.1.10 <testi@example.com>: Recipient address rejected: Domain example.com does not accept mail (nullMX)
D/SmtpTransport(30906): SMTP <<< 354 End data with <CR><LF>.<CR><LF>
D/SmtpTransport(30906): SMTP >>> .
D/SmtpTransport(30906): SMTP <<< 250 2.0.0 Ok: queued as C4F2A108F
D/SmtpTransport(30906): SMTP >>> QUIT
D/SmtpTransport(30906): SMTP <<< 221 2.0.0 Bye
E/MessagingController(30906): Failed to send message
E/MessagingController(30906): com.fsck.k9.mail.transport.smtp.EnhancedNegativeSmtpReplyException: <testi@example.com>: Recipient address rejected: Domain example.com does not accept mail (nullMX)
E/MessagingController(30906):   at com.fsck.k9.mail.transport.smtp.SmtpTransport.buildEnhancedNegativeSmtpReplyException(SmtpTransport.java:632)
E/MessagingController(30906):   at com.fsck.k9.mail.transport.smtp.SmtpTransport.responseLineToCommandResponse(SmtpTransport.java:711)
E/MessagingController(30906):   at com.fsck.k9.mail.transport.smtp.SmtpTransport.readPipelinedResponse(SmtpTransport.java:671)
E/MessagingController(30906):   at com.fsck.k9.mail.transport.smtp.SmtpTransport.sendMessageTo(SmtpTransport.java:460)
E/MessagingController(30906):   at com.fsck.k9.mail.transport.smtp.SmtpTransport.sendMessage(SmtpTransport.java:428)
E/MessagingController(30906):   at com.fsck.k9.controller.MessagingController.sendPendingMessagesSynchronous(MessagingController.java:2630)
E/MessagingController(30906):   at com.fsck.k9.controller.MessagingController$21.run(MessagingController.java:2514)
E/MessagingController(30906):   at com.fsck.k9.controller.MessagingController.runInBackground(MessagingController.java:206)
E/MessagingController(30906):   at com.fsck.k9.controller.MessagingController.access$000(MessagingController.java:122)
E/MessagingController(30906):   at com.fsck.k9.controller.MessagingController$1.run(MessagingController.java:175)
E/MessagingController(30906):   at java.lang.Thread.run(Thread.java:818)

[Fjorstut]

K-9 Mail version 5.600 Please consider to fix this issue. I receive 'undeliverable mail' (*) bounce back messages when trying to send to many recipients. I receive bounce back messages from only gmail and yahoo recipients.

yahoo says: 554 Message not allowed - [299] gmail says: 550 5.7.1 [ip] Our system has detected that this message is not RFC 5322 compliant: 'From' header is missing.

[popindavibe]

@Fjorstut if you administer the mail server, a workaround (until this is addressed) is to lower the reject rules for unknown recipient domain and non fqdn recipient.

On postfix this is what I set up:

smtpd_recipient_restrictions =
        reject_unauth_pipelining, 
        ## K9 Mail bug 
        ##reject_unknown_recipient_domain, 
        ##reject_non_fqdn_recipient, 
        check_policy_service inet:localhost:9999, 
        permit_sasl_authenticated, 
        reject_non_fqdn_recipient, 
        reject_unknown_recipient_domain,
[..]

[Fjorstut]

@Fjorstut if you administer the mail server, a workaround (until this is addressed) is to lower the reject rules for unknown recipient domain and non fqdn recipient.

On postfix this is what I set up:

smtpd_recipient_restrictions =
        reject_unauth_pipelining, 
        ## K9 Mail bug 
        ##reject_unknown_recipient_domain, 
        ##reject_non_fqdn_recipient, 
        check_policy_service inet:localhost:9999, 
        permit_sasl_authenticated, 
        reject_non_fqdn_recipient, 
        reject_unknown_recipient_domain,
[..]

Thanks a lot for your tip, @popindavibe , much appreciated. It is a nuisance K-9 annoyance. I ended up using Outlook client on PC instead.

[asaivan]

Hello, K-9 team,

I want to also chime in on this as this is an issue that we've been seeing in K-9 for years now, and have only just now been able to pin down the cause of the problem and we can confirm that it's indeed issue as described above. K-9 doesn't handle rejects set by Postfix in a graceful way, and so we have emails with blank bodies sent out, resulting in confusion and messages not going through properly. For our organization this is a critical issue and we hope that this problem can be addressed soon.

Thank you for your attention to this matter.

[popindavibe]

Coming here after a long time, is there any news on this?