search

apple / ccs-calendarserver

The Calendar and Contacts Server.
https://www.calendarserver.org
Apache License 2.0
486 stars 136 forks source link

replies to tokenized mails are not forwared correctly #324

Closed macosforgebot closed 13 years ago

macosforgebot commented 13 years ago

mail@… originally submitted this as ticket:416

Hi,

when replying to a tokenized mail, the message should be forwared. This does not work with r6910.

The header is included twice in the mail send to the organizer:

Delivered-To: felixjmoeller@googlemail.com
Received: by 10.42.5.84 with SMTP id 20cs50393icv;
        Wed, 16 Feb 2011 13:19:59 -0800 (PST)
Received: by 10.204.121.73 with SMTP id g9mr927404bkr.37.1297891198262;
        Wed, 16 Feb 2011 13:19:58 -0800 (PST)
Return-Path: <felix@derklecks.de>
Received: from mx05.ispgateway.de (mx05.ispgateway.de [80.67.18.102])
        by mx.google.com with ESMTP id w2si525387bkw.2.2011.02.16.13.19.56;
        Wed, 16 Feb 2011 13:19:57 -0800 (PST)
Received-SPF: neutral (google.com: 80.67.18.102 is neither permitted nor denied by best guess record for domain of felix@derklecks.de) client-ip=80.67.18.102;
Authentication-Results: mx.google.com; spf=neutral (google.com: 80.67.18.102 is neither permitted nor denied by best guess record for domain of felix@derklecks.de) smtp.mail=felix@derklecks.de
X-Envelope-To: mail@felixmoeller.de
Received: from [80.67.31.39] (helo=smtprelay01.ispgateway.de)
    by mx05.ispgateway.de with esmtp (Exim 4.68)
    (envelope-from <felix@derklecks.de>)
    id 1Ppomy-0001r3-EA
    for mail@felixmoeller.de; Wed, 16 Feb 2011 22:19:56 +0100
Received: from [87.118.84.209] (helo=km30208-01.keymachine.de)
    by smtprelay01.ispgateway.de with esmtpsa (TLSv1:AES256-SHA:256)
    (Exim 4.68)
    (envelope-from <felix@derklecks.de>)
    id 1Ppomy-0001xO-85
    for mail@felixmoeller.de; Wed, 16 Feb 2011 22:19:56 +0100
Message-Id: <E1Ppomy-0001xO-85@smtprelay01.ispgateway.de>
From: felix@derklecks.de
Date: Wed, 16 Feb 2011 22:19:56 +0100
X-Df-Sender: caldav@felixmoeller.de
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
    spamfilter17.ispgateway.de
X-Spam-Level: 
X-Spam-Status: No, hits=-6.6 required=9999.0 tests=BAYES_00 autolearn=disabled
    version=3.2.5
X-Spam-CMAETAG: v=1.1 cv=cMlJ0YU4b11YdzE+9VDlRaASnYjgXZNeOWRfzO9tuzM=
    c=1 sm=0 a=UCyEHwDE5oLcrrS2avgM0g==:17 a=9H6MWKLwAAAA:8
    a=0c5oy00WbJ3t26eDUD4A:9 a=9FxSG3nE73km3IToEnpDMJDl2UAA:4
    a=RiGi7OWhmSoA:10 a=0rt2_-LiRAYA:10 a=HKIhatoINoYA:10 a=wK4z2fZwwS4A:10
    a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 xcat=Undefined/Undefined
X-Spam-CMAECATEGORY: 0
X-Spam-CMAESUBCATEGORY: 0
X-Spam-CMAESCORE: 0

From nobody Wed Feb 16 22:19:51 2011
Return-Path: <felix@derklecks.de>
Received: from compute2.internal (compute2.nyi.mail.srv.osa [10.202.2.42])
    by store71 (Cyrus git2.5.0+0-git-fastmail-6410) with LMTPA;
    Wed, 16 Feb 2011 16:19:51 -0500
X-Sieve: CMU Sieve 2.4
X-Spam-charsets: from='UTF-8', plain='UTF-8'
X-Resolved-to: felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm
X-Delivered-to: felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm
X-Mail-from: felix@derklecks.de
Received: from mx1.messagingengine.com ([10.202.2.200])
    by compute2.internal (LMTPProxy); Wed, 16 Feb 2011 16:19:51 -0500
Received: from smtprelay01.ispgateway.de (smtprelay01.ispgateway.de
    [80.67.31.24])
    by mx1.messagingengine.com (Postfix) with ESMTP id 32C59960108
    for <felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm>;
    Wed, 16 Feb 2011 16:19:50 -0500 (EST)
Received: from [92.201.59.107] (helo=[192.168.178.22])
    by smtprelay01.ispgateway.de with esmtpsa (TLSv1:AES256-SHA:256)
    (Exim 4.68) (envelope-from <felix@derklecks.de>) id 1Ppomq-0001ug-Hf
    for felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm;
    Wed, 16 Feb 2011 22:19:48 +0100
Message-ID: <4D5C3F73.1040608@derklecks.de>
Date: Wed, 16 Feb 2011 22:19:47 +0100
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
    rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc14
    Lightning/1.0b3pre Thunderbird/3.1.7
MIME-Version: 1.0
Subject: Re: Aktualisierung: New Event
References: <20110216211842.28004.188035797.1@km30208-01.keymachine.de>
In-Reply-To: <20110216211842.28004.188035797.1@km30208-01.keymachine.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Df-Sender: mail@felixmoeller.de
X-Truedomain-Domain: derklecks.de
X-Truedomain-SPF: No Record
X-Truedomain-DKIM: No Signature
X-Truedomain-ID: D986B5704FFB8CC79811EDDC3DE8254C
X-Truedomain: Neutral
From: felix@derklecks.de
Reply-To: felix@derklecks.de
To: mail@felixmoeller.de

asdasdasd

Am 16.02.2011 22:18, schrieb Felix Moeller:

> Aktualisierung

>

>

>   New Event

>

>

>       Veranstalter:

>

> Felix Moeller <mailto:mail@felixmoeller.de>

>

>

>       Ort:

>

>

>       Datum:

>

> Wednesday, 16. February 2011

>

>

>       Zeit:

>

> 11:00 abends bis 12:00 morgens CET (1 Stunde)

>

>

>       Beschreibung:

>

> Dies ist ein toller Test Ich habe keine ahnung, was das nun wird Wieder

> eine Zeile.

>

>

>       Teilnehmer:

>

With the following patch, to strip out the envelope header:

# diff -u /usr/lib/python2.6/site-packages/twistedcaldav/mail.py{.orig,}
--- /usr/lib/python2.6/site-packages/twistedcaldav/mail.py.orig 2011-02-16 23:48:31.000000000 +0100
+++ /usr/lib/python2.6/site-packages/twistedcaldav/mail.py      2011-02-16 23:51:31.000000000 +0100
@@ -750,7 +750,7 @@
             msg["To"] = toAddr
             factory = ESMTPSenderFactory(
                 settings["Username"], settings["Password"],
-                fromAddr, toAddr, StringIO(str(msg)), deferred,
+                fromAddr, toAddr, StringIO(msg.as_string()), deferred,
                 contextFactory=contextFactory,
                 requireAuthentication=False,
                 requireTransportSecurity=settings["UseSSL"],

One gets the following:

Delivered-To: felixjmoeller@googlemail.com
Received: by 10.42.5.84 with SMTP id 20cs54326icv;
        Wed, 16 Feb 2011 14:53:30 -0800 (PST)
Received: by 10.204.114.130 with SMTP id e2mr995481bkq.54.1297896809204;
        Wed, 16 Feb 2011 14:53:29 -0800 (PST)
Return-Path: <felix@derklecks.de>
Received: from mx10.ispgateway.de (mx10.ispgateway.de [80.67.29.31])
        by mx.google.com with ESMTP id 27si712366bkl.99.2011.02.16.14.53.26;
        Wed, 16 Feb 2011 14:53:28 -0800 (PST)
Received-SPF: neutral (google.com: 80.67.29.31 is neither permitted nor denied by best guess record for domain of felix@derklecks.de) client-ip=80.67.29.31;
Authentication-Results: mx.google.com; spf=neutral (google.com: 80.67.29.31 is neither permitted nor denied by best guess record for domain of felix@derklecks.de) smtp.mail=felix@derklecks.de
X-Envelope-To: mail@felixmoeller.de
Received: from [80.67.31.95] (helo=smtprelay06.ispgateway.de)
    by mx10.ispgateway.de with esmtp (Exim 4.68)
    (envelope-from <felix@derklecks.de>)
    id 1PpqFS-0007AN-Nq
    for mail@felixmoeller.de; Wed, 16 Feb 2011 23:53:26 +0100
Received: from [87.118.84.209] (helo=km30208-01.keymachine.de)
    by smtprelay06.ispgateway.de with esmtpsa (TLSv1:AES256-SHA:256)
    (Exim 4.68)
    (envelope-from <felix@derklecks.de>)
    id 1PpqFS-0005DO-JG
    for mail@felixmoeller.de; Wed, 16 Feb 2011 23:53:26 +0100
Received: from compute2.internal (compute2.nyi.mail.srv.osa [10.202.2.42])
    by store71 (Cyrus git2.5.0+0-git-fastmail-6410) with LMTPA;
    Wed, 16 Feb 2011 17:53:15 -0500
X-Sieve: CMU Sieve 2.4
X-Spam-charsets: from='UTF-8', plain='UTF-8'
X-Resolved-to: felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm
X-Delivered-to: felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm
X-Mail-from: felix@derklecks.de
Received: from mx1.messagingengine.com ([10.202.2.200])
    by compute2.internal (LMTPProxy); Wed, 16 Feb 2011 17:53:15 -0500
Received: from smtprelay02.ispgateway.de (smtprelay02.ispgateway.de
    [80.67.18.14])
    by mx1.messagingengine.com (Postfix) with ESMTP id 791DD9601A0
    for <felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm>;
    Wed, 16 Feb 2011 17:53:14 -0500 (EST)
Received: from [92.201.59.107] (helo=[192.168.178.22])
    by smtprelay02.ispgateway.de with esmtpsa (TLSv1:AES256-SHA:256)
    (Exim 4.68) (envelope-from <felix@derklecks.de>) id 1PpqFE-0001jI-On
    for felixcaldav+f70331ea-93eb-46c1-b2ef-242e497b245d@fastmail.fm;
    Wed, 16 Feb 2011 23:53:12 +0100
Message-ID: <4D5C5558.5000808@derklecks.de>
Date: Wed, 16 Feb 2011 23:53:12 +0100
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
    rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc14
    Lightning/1.0b3pre Thunderbird/3.1.7
MIME-Version: 1.0
Subject: Re: Aktualisierung: New Event
References: <20110216211842.28004.188035797.1@km30208-01.keymachine.de>
In-Reply-To: <20110216211842.28004.188035797.1@km30208-01.keymachine.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
X-Df-Sender: mail@felixmoeller.de
X-Truedomain-Domain: derklecks.de
X-Truedomain-SPF: No Record
X-Truedomain-DKIM: No Signature
X-Truedomain-ID: A2D14D626AF708F4D1D64ABCC6450B83
X-Truedomain: Neutral
From: felix@derklecks.de
Reply-To: felix@derklecks.de
To: mail@felixmoeller.de
X-Df-Sender: caldav@felixmoeller.de
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on
    spamfilter20.ispgateway.de
X-Spam-Level: 
X-Spam-Status: No, hits=-1.0 required=9999.0 tests=BAYES_40 autolearn=disabled
    version=3.2.5
X-Spam-CMAETAG: v=1.1 cv=ZU3YeN+3KNy97QZ3rp+vns8tjEJ2nYA8q82V91AbUOk=
    c=1 sm=0 a=fKqi_JUXJ1EA:10 a=IkcTkHD0fZMA:10
    a=g/tIsaK+EP3zO1Fe16Z+5g==:17 a=KiBad21AOLZ3IfF_tUIA:9
    a=0mXqJaMQZH6zeTUb6R8A:7 a=dGNYB49sXqqeb-LLMdf8Plo_H50A:4
    a=QEXdDO2ut3YA:10 a=wK4z2fZwwS4A:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117
    xcat=Undefined/Undefined
X-Spam-CMAECATEGORY: 0
X-Spam-CMAESUBCATEGORY: 0
X-Spam-CMAESCORE: 0

asd

Am 16.02.2011 22:18, schrieb Felix Moeller:

> Aktualisierung

>

>

>   New Event

>

>

>       Veranstalter:

>

> Felix Moeller <mailto:mail@felixmoeller.de>

>

>

>       Ort:

>

>

>       Datum:

>

> Wednesday, 16. February 2011

>

>

>       Zeit:

>

> 11:00 abends bis 12:00 morgens CET (1 Stunde)

>

>

>       Beschreibung:

>

> Dies ist ein toller Test Ich habe keine ahnung, was das nun wird Wieder

> eine Zeile.

>

>

>       Teilnehmer:

>
macosforgebot commented 13 years ago

mail@… originally submitted this as comment:1:⁠ticket:416

My patch in the initial comment does not solve this issue entirely as somewhere empty lines are inserted -- as can be seen in the output above.

But it at least allows one to read the mail.

macosforgebot commented 13 years ago

@m0rgen originally submitted this as comment:2:⁠ticket:416

I am not able to reproduce this using OS X Server. I tested by having Calendar Server send an iMIP invite to a gmail account, and then instead of using the iMIP mechanism within gmail to respond, I hit the Reply button and typed in some text without including the .ics attachment. This caused Calendar Server to forward that message to the original organizer of the event (with no header duplication).

macosforgebot commented 13 years ago

mail@… originally submitted this as attachment:issue416.py:⁠ticket:416

test case to see duplicating headers

macosforgebot commented 13 years ago

mail@… originally submitted this as attachment:calendarserver-mailforwarding.patch:⁠ticket:416

patch to fix all mail forwarding issues for me

macosforgebot commented 13 years ago

mail@… originally submitted this as attachment:mail.tar.gz:⁠ticket:416

here you see the diffrent mailstrings dumped into text files

macosforgebot commented 13 years ago

mail@… originally submitted this as attachment:fixlines.patch:⁠ticket:416

the attached dumps have been generated by the following patch

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:3:⁠ticket:416

I spend a long time debugging this issue.

This issue is caused by different linebreaks: It was caused by somehow incorrect lineending handling.

Read from my IMAP, I have message containing \r\n when they get read by message_from_string() They are delimited by \n in the header and still \r\n in the content.

Sending the mail via ESMTPSenderFactory() now creates two \n into \r\n as of RFC2822, but does so for the content too, therefore too many line breaks are in there.

One MUST NOT use str(msg) as this contains the unix header (envelope) and therefore the headers will be added twice ...

My issue416.py is useless and wrong. But calendarserver-mailforwarding.patch solves all my problems.

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:4:⁠ticket:416

@sagen: thanks alot for testing this! I hope with the info attached you see my issue.

For the double spaced lines one has to send a plain/text message, otherwise the linebreaks will not be seen...

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:5:⁠ticket:416

Could somebody look at calendarserver-mailforwarding.patch, I still need it.

macosforgebot commented 13 years ago

@m0rgen originally submitted this as comment:6:⁠ticket:416

Sorry for the delay, how's this: https://trac.calendarserver.org/changeset/7819/CalendarServer/trunk

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:7:⁠ticket:416

Thank you. This works for me.