Exim: Failed to get envelope sender in "Return-path:" field

[dpc22]

Version

6.2.66

Installation method

My own rpm, derived from the "official" RHEL 7 rpms

Expected behavior

If someone sends a message to a moderated list, they receive a notification message as you would expect:

Your message to the list 'XXX' has been forwarded to the moderator(s)

If they have also set up an out of office message on the mailbox which receives this notification then (at least when using Exchange Online) this out of office message is send to sympa@domain with an empty SMTP return path (the "<>" below):

2022-02-24 10:09:06 +0000 1nNB3a-0008az-Hh
        <= <>
        H=ppsw-30.csi.cam.ac.uk [131.111.8.130]:50470
        I=[131.111.8.15]:25 P=esmtp S=12985 
        for sympa@lists.cam.ac.uk

I would not expect this to generate any further messages given the potential for confusion and mail loops.

Actual behavior

Sympa appears to fish a sending email address from the out of office message and complains:

This is an automatic response sent by Sympa Mailing Lists Manager.
Impossible to distribute your message for list sympa@lists.cam.ac.uk for the
following reason:

No command found in message.

I think that Sympa should test for the empty return path here before it attempts to process the message as a list of commands.

Feb 24 10:09:08 lists-1 sympa_msg[32474]: notice Sympa::Spindle::ProcessIncoming::_twist() Processing Sympa::Message [<sympa@lists.cam.ac.uk.1645697346.33048>](mailto:sympa@lists.cam.ac.uk.1645697346.33048); envelope_sender=; [message_id=XXX@CWLP265MB4094.GBRP265.PROD.OUTLOOK.COM](mailto:message_id=XXX@CWLP265MB4094.GBRP265.PROD.OUTLOOK.COM); [sender=YYY@fitz.cam.ac.uk](mailto:sender=YYY@fitz.cam.ac.uk)
Feb 24 10:09:08 lists-1 sympa_msg[32474]: notice Sympa::Request::Message::__parse() Parsing: Automatic reply: Message distribution
Feb 24 10:09:08 lists-1 sympa_msg[32474]: notice Sympa::Request::Message::__parse() Parsing: <...the text of the out of office message...>
Feb 24 10:09:08 lists-1 sympa_msg[32474]: info Sympa::Spindle::DoCommand::_twist() No command found in message Sympa::Message [<sympa@lists.cam.ac.uk.1645697346.33048/shelved:arc_cv=pass>](mailto:sympa@lists.cam.ac.uk.1645697346.33048/shelved:arc_cv=pass)

[ikedas]

Hi @dpc22 ,

Could you please show us the content (at least the subject and the body) of that " out of office message" ?

[dpc22]

I don't have a copy of the subject line. The out of office was generated by Exchange Online, so it will probably be of the form:

Automatic reply: $original_subject

so:

Automatic reply: Message distribution

The message content is recorded in the Sympa logs:

I am working reduced hours whilst I recover from being unwell.
Therefore there may be a delay in responding to emails. Any urgent
Fitzwilliam College IT and AV related problems or requests should be
sent to XXX@fitz.cam.ac.uk) or filed as a helpdesk ticket at
https://help.fitz.cam.ac.uk/

[ikedas]

As a workaround, you may disable sympa@mydomain, i.e. removing this address from the setting of MTA.

[dpc22]

Don't people have to email sympa@mydomain in order to send email commands to the list? For example:

From: David Carter dpc22@cam.ac.uk To: SYMPA sympa@lists.cam.ac.uk Subject: help

responses:

             SYMPA -- Systeme de Multi-Postage Automatique
                       (Automatic Mailing System)

                                User's Guide

SYMPA is an electronic mailing-list manager that automates list management
functions such as subscriptions, moderation, and archive management.
[...]

[ikedas]

Don't people have to email sympa@mydomain in order to send email commands to the list? For example:

Exactly. The definition for this email address is included in setting of MTA as an alias, and may be removed if none uses mail commands.

[ikedas]

Hi @dpc22 ,

It turned out that, as of Sympa 6.2.13 or later, the response "No command found in message." was sent as DSN by Sympa (cf. the commit). Sympa won't send DSN as a response to the incoming message with null envelope sender.

Additionally, Sympa won't process the message with null envelope sender: Instead it will be ignored with this log:

Oct 30 16:09:50 hostname sympa_msg[2017]: notice Sympa::Spindle::DoCommand::_twist() Sympa::Message <sympa@mail.example.org.1667113788.2298>: Ignoring message which would cause a loop; message appears to be DSN report

Thus, there seem no problem.

[ikedas]

Anyways it is worth sending the first response "Your message to the list 'XXX' has been forwarded to the moderator(s)" as an DSN: Legitimate autoresponders won't respond to DSNs. Please check the PR above.

[dpc22]

I'm puzzled by:

Sympa won't send DSN as a response to the incoming message with null envelope sender

as Sympa 6.2.66 was definitely generating these "No command found in message" responses to out of office messages with null envelope sender earlier in the year.

Does (#1508) change the behaviour in some way? It isn't immediately clear to me what that does. Thanks.

[ikedas]

I'm puzzled by:

Sympa won't send DSN as a response to the incoming message with null envelope sender

as Sympa 6.2.66 was definitely generating these "No command found in message" responses to out of office messages with null envelope sender earlier in the year.

Sympa 6.2.66 rejects such messages: https://github.com/sympa-community/sympa/blob/b17e3bb89b904d90704978afe3b9e24bb32f4cf1/src/lib/Sympa/Spindle/DoCommand.pm#L72-L81

What is possible is that that out of office message did not have the null envelope sender.

Does (#1508) change the behaviour in some way? It isn't immediately clear to me what that does. Thanks.

No. This PR changes the first message described in above to the DSN. Thus auto-responder will not respond to the messages, i.e. out of office message will not be sent.

[dpc22]

The log entry that I quoted when I opened the ticket indicated a null envelope sender (the <= <> bit below):

2022-02-24 10:09:06 +0000 1nNB3a-0008az-Hh
        <= <>
        H=ppsw-30.csi.cam.ac.uk [131.111.8.130]:50470
        I=[131.111.8.15]:25 P=esmtp S=12985 
        for sympa@lists.cam.ac.uk

I guess that it is possible that something else is going on here. I will do some testing.

[dpc22]

The following SMTP dialogue generates a "No command found in message" response to dpc22@cam.c.uk with Sympa 6.2.68:

$ telnet mx.cam.ac.uk smtp
Trying 131.111.8.147...
Connected to mx.CAM.AC.UK.
Escape character is '^]'.
220 ppsw-31.srv.uis.cam.ac.uk (mx.cam.ac.uk [131.111.8.147]:25) ESMTP Exim 4.96-0 Mon, 31 Oct 2022 08:46:23 +0000

HELO me
250 ppsw-31.srv.uis.cam.ac.uk Hello user-5.uis.private.cam.ac.uk [172.16.56.5]

MAIL FROM:<>
250 OK

RCPT TO:<sympa@lists.cam.ac.uk>
250 Accepted

DATA
354 Enter message, ending with "." on a line by itself
From: dpc22@cam.ac.uk
To: sympa@lists.cam.ac.uk
Subject: Test
Message-ID: 12345@localhost

Test

.

You should be able to replicate that. I don't think that I have made any local changes which would affect this behaviour.

[ikedas]

Please show the log of Sympa from when the maili server accepted the message to when Sympa responded (or didn't respond) to that message.

[dpc22]

2022-10-31 08:47:15 +0000 1opQRh-0006XN-0v
        <= <>
        H=ppsw-31.srv.uis.cam.ac.uk [131.111.8.131]:48150
        I=[131.111.8.15]:25 P=esmtp S=1481
        id=12345@localhost
        for sympa@lists.cam.ac.uk
2022-10-31 08:47:16 +0000 1opQRh-0006XN-0v
        => sympa <sympa@lists.cam.ac.uk>
        F=<> P=<>
        R=sympa_command T=sympa_command S=1526 QT=1s DT=1s

Oct 31 08:47:18 lists-1 sympa_msg[2923]: notice Sympa::Spindle::ProcessIncoming::_twist() Processing Sympa::Message <sympa@lists.cam.ac.uk.1667206035.25136>; envelope_sender=; message_id=12345@localhost; sender=dpc22@cam.ac.uk
Oct 31 08:47:18 lists-1 sympa_msg[2923]: notice Sympa::Request::Message::__parse() Parsing: Test
Oct 31 08:47:18 lists-1 sympa_msg[2923]: notice Sympa::Request::Message::__parse() Parsing: Test
Oct 31 08:47:18 lists-1 sympa_msg[2923]: info Sympa::Spindle::DoCommand::_twist() No command found in message Sympa::Message <sympa@lists.cam.ac.uk.1667206035.25136/shelved:arc_cv=none>
Oct 31 08:47:18 lists-1 sympa_msg[2923]: notice Sympa::Spindle::ProcessTemplate::_twist() Processing Sympa::Message::Template <sympa@lists.cam.ac.uk.1667206038.2923,359/shelved:dkim_sign>; envelope_sender=<>; message_id=sympa.1667206038.732132.2923.414@lists.cam.ac.uk; recipients=dpc22@cam.ac.uk; sender=sympa-request@lists.cam.ac.uk; template=delivery_status_notification; action=failed; status=5.6.1
Oct 31 08:47:18 lists-1 sympa_msg[2923]: notice Sympa::Spool::Outgoing::store() Message Sympa::Message::Template <sympa@lists.cam.ac.uk.1667206038.2923,359/shelved:dkim_sign> is stored into bulk spool as <1.5.1667206038.1667206038.742006.sympa@lists.cam.ac.uk_s,2923,8065>
Oct 31 08:47:18 lists-1 bulk[7545]: notice Sympa::Spindle::ProcessOutgoing::_twist() Start sending message Sympa::Message <1.5.1667206038.1667206038.742006.sympa@lists.cam.ac.uk_s,2923,8065/s/shelved:dkim_sign> to lists.cam.ac.uk (priority 1) (starting 0 seconds after scheduled expedition date)

2022-10-31 08:47:19 +0000 1opQRy-0006Xc-31
        <= <> U=sympa P=local S=3724
        id=sympa.1667206038.732132.2923.414@lists.cam.ac.uk
        for dpc22@cam.ac.uk
2022-10-31 08:47:19 +0000 1opQRy-0006Xc-31
        => dpc22@cam.ac.uk
        F=<> P=<>
        R=remote T=smtp S=3823
        H=ppsw.cam.ac.uk [131.111.8.137]
        I=[131.111.8.13]
        C="250 OK
        id=1opQRz-0007Ad-X3" QT=0s DT=0s```

[ikedas]

Oct 31 08:47:18 lists-1 sympa_msg[2923]: notice Sympa::Spindle::ProcessIncoming::_twist() Processing Sympa::Message <sympa@lists.cam.ac.uk.1667206035.25136>; envelope_sender=; message_id=12345@localhost; sender=dpc22@cam.ac.uk

Envelope sender isn't passed from Postfix to Sympa. Expected log line is such as:

Oct 31 17:59:47 hostname sympa_msg[2017]: notice Sympa::Spindle::ProcessIncoming::_twist() Processing Sympa::Message <sympa@mail.example.org.1667206786.30781>; envelope_sender=<>; message_id=12345@localhost; sender=user@example.com

You probably set up Postfix with "virtual domain setting" described in the manual. If you did, review the setting in master.cf

[dpc22]

I am running Exim.

Wouldn't "envelope_sender=" indicate an empty envelope sender address? "<>" is just quoting around the actual value.

That certainly appears to be true of other email addresses which are quoted in the Sympa logs. There are never any "<" or ">" after the "envelope_sender=" clauses in my logs. Example: envelope_sender=dpc22@cam.ac.uk

[ikedas]

Oh sorry.

On Exim, pipe transport(s) for Sympa should have return_path_add. Check the manual about Exim 4.

[dpc22]

Aha. If it is looking for a "Return-Path: " header in the message body that would explain things.

[dpc22]

Okay, I added return_path_add to my Exim configuration. That didn't fix the problem, although I agree that it should.

After a lot of head scratching, I think that this is just because the header that Exim adds to the message is "Return-path" rather than "Return-Path" (note the case of "path"). The Exim source contains:

src/transport.c:    uschar * s = string_sprintf("Return-path: <%.*s>\n%n",

If I log "$serialized" in Sympa/Message.pm I see:

X-Sympa-To: sympa@lists.cam.ac.uk
Return-path: <>
Received: from [127.0.0.1] (port=57784 helo=me)
        by lists-2.srv.uis.cam.ac.uk (lists.cam.ac.uk [127.0.0.1]:25)
        with smtp id 1opTQc-0000n4-34 (Exim 4.96) for sympa@lists.cam.ac.uk
        (return-path <>);
        Mon, 31 Oct 2022 11:58:18 +0000
From: dpc22@cam.ac.uk
To: sympa@lists.cam.ac.uk
Subject: Test
Message-ID: 3@localhost

Test

That doesn't match the following in Sympa/Message.pm:

   if ($serialized =~ /\GReturn-Path: (.*?)\n(?![ \t])/cgs
        and not exists $self->{'envelope_sender'}) {

I confess that I don't know what "\G" does in Perl regular expressions, but "Return-Path" rather than "Return-path" seems to be significant. Presumably we can use just add an "i" modifier to the end of "/cgs"

[ikedas]

@dpc22, thank you for investigation. The PR above will fix the problem.