apple / ccs-calendarserver

The Calendar and Contacts Server.
https://www.calendarserver.org
Apache License 2.0
486 stars 136 forks source link

line break in event gets send back as \\n #322

Closed macosforgebot closed 13 years ago

macosforgebot commented 13 years ago

mail@… originally submitted this as ticket:414


I have reproduced this with an iPhone and with Lightning.

PUT /calendars/users/felix/calendar/eec52fe2-737e-4edb-8b90-6e87c064bae8.ics
HTTP/1.1
Host: www.felixmoeller.de:8008
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101209
Fedora/3.1.7-0.35.b3pre.fc14 Lightning/1.0b3pre Thunderbird/3.1.7
Accept: text/xml
Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
Accept-Encoding: gzip,deflate
Accept-Charset: utf-8,*;q=0.1
Keep-Alive: 115
Connection: keep-alive
Content-Length: 797
Content-Type: text/calendar; charset=utf-8
If-Match: "094613aa1efc7f1b1a33b1394645aa50"
Authorization: Digest username="asd", realm="Test Realm",
nonce="938714704201974679450742567",
uri="/calendars/users/felix/calendar/eec52fe2-737e-4edb-8b90-6e87c064bae8.ics",
algorithm=MD5, response="52bf7e6df238af6f4ba2e6a20f321680"
Pragma: no-cache
Cache-Control: no-cache

BEGIN:VCALENDAR
PRODID:-//Mozilla.org/NONSGML Mozilla Calendar V1.1//EN
VERSION:2.0
BEGIN:VTIMEZONE
TZID:Europe/Berlin
X-LIC-LOCATION:Europe/Berlin
BEGIN:DAYLIGHT
TZOFFSETFROM:+0100
TZOFFSETTO:+0200
TZNAME:CEST
DTSTART:19700329T020000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=3
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:+0200
TZOFFSETTO:+0100
TZNAME:CET
DTSTART:19701025T030000
RRULE:FREQ=YEARLY;BYDAY=-1SU;BYMONTH=10
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
CREATED:20110207T222327Z
LAST-MODIFIED:20110207T231033Z
DTSTAMP:20110207T231033Z
UID:eec52fe2-737e-4edb-8b90-6e87c064bae8
SUMMARY:Neuer Termin
DTSTART;TZID=Europe/Berlin:20110207T230000
DTEND;TZID=Europe/Berlin:20110208T000000
DESCRIPTION:1\n2\n3\n\n5
TRANSP:OPAQUE
X-MOZ-GENERATION:2
END:VEVENT
END:VCALENDAR

Response:

HTTP/1.1 204 No Content
Content-Length: 0
Server: CalendarServer/3.0(rexportiert) Twisted/10.1.0 TwistedWeb/9.0.0
Last-Modified: Mon, 07 Feb 2011 22:40:24 GMT
DAV: 1, access-control, calendar-access, calendar-schedule,
calendar-auto-schedule, calendar-availability, inbox-availability,
calendar-proxy, calendarserver-private-events, calendarserver-private-comments,
calendarserver-sharing, calendarserver-sharing-no-scheduling, addressbook,
calendarserver-principal-property-search
ETag: "8301ddf72fc95e0080a82b7ca271db5c"
Date: Mon, 07 Feb 2011 23:10:29 GMT

looking in the database, the event has \\n and accessing the vcard via the webinterface results in \\n too.

Which path does this data take and where could it be modified?

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:1:⁠ticket:414


I have one more information: The mail that is send out to other attendees still has the correct line breaks. So they get reset somewhere later...

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:2:⁠ticket:414


Another finding is that the double on each save. This means when saving \\n I get \\\\n} back and it continues.

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:3:⁠ticket:414


it is not just the VEVENT it seems to happen in every VOBJECT (I tried the VTODO).

I double checked that I am using vobject library as of revision 219 and the three patches from source:/CalendarServer/trunk/lib-patches/vobject/ are applied.

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:4:⁠ticket:414


I tried to debug this yesterday on IRC with dre.

Testing this with iCal is somehow difficult. Entering an event with iCal the line breaks stay correct, BUT enabling the HTTP debugging in iCal I think the even is not even read again. I can see the PUT Request, but then just the ETag is compared and the event is not reread from the server.

Looking at an iCal created event from Lightning or iOS shows again the \n in the text ...

macosforgebot commented 13 years ago

@wsanchez originally submitted this as comment:5:⁠ticket:414

macosforgebot commented 13 years ago

@wsanchez originally submitted this as comment:6:⁠ticket:414

macosforgebot commented 13 years ago

@dreness originally submitted this as comment:7:⁠ticket:414


Replying to mail@…:

I tried to debug this yesterday on IRC with dre.

Testing this with iCal is somehow difficult. Entering an event with iCal the line breaks stay correct, BUT enabling the HTTP debugging in iCal I think the even is not even read again. I can see the PUT Request, but then just the ETag is compared and the event is not reread from the server.

Looking at an iCal created event from Lightning or iOS shows again the \n in the text ...

New description:

(sorry for the spam, my first attempt at replying was wrong so I had to roll the ticket back)

I'm not able to reproduce this using current trunk (r6917), iCal (Version 4.0.4 (1395) in Mac OS X 10.6.5, and iOS 4.2.1. Creating and event with iCal and iOS both seem to produce the same correct behavior, and I'm also able to edit the events later without seeing any additional unexpected line breaks.

A multi-line description is displayed as followed by CalDAVClientLibrary:

DESCRIPTION:First line\nSecond line\nThird line added later.

Looking in the DB, it also appears correct:

caldav=# select icalendar_text from calendar_object;
                         icalendar_text
----------------------------------------------------------------
 BEGIN:VCALENDAR\r                                             +
 VERSION:2.0\r                                                 +
 CALSCALE:GREGORIAN\r                                          +
 PRODID:-//Apple Inc.//iCal 4.0.4//EN\r                        +
 BEGIN:VTIMEZONE\r                                             +
 TZID:America/Los_Angeles\r                                    +
 BEGIN:STANDARD\r                                              +
 DTSTART:20071104T020000\r                                     +
 RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU\r                      +
 TZNAME:PST\r                                                  +
 TZOFFSETFROM:-0700\r                                          +
 TZOFFSETTO:-0800\r                                            +
 END:STANDARD\r                                                +
 BEGIN:DAYLIGHT\r                                              +
 DTSTART:20070311T020000\r                                     +
 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU\r                       +
 TZNAME:PDT\r                                                  +
 TZOFFSETFROM:-0800\r                                          +
 TZOFFSETTO:-0700\r                                            +
 END:DAYLIGHT\r                                                +
 END:VTIMEZONE\r                                               +
 BEGIN:VEVENT\r                                                +
 UID:CDA73304-180F-48A7-9078-643993E715E5\r                    +
 DTSTART;TZID=America/Los_Angeles:20110214T120000\r            +
 DTEND;TZID=America/Los_Angeles:20110214T130000\r              +
 CREATED:20110214T191732Z\r                                    +
 DESCRIPTION:this is a note\nand another line\r                +
 DTSTAMP:20110214T201324Z\r                                    +
 SEQUENCE:4\r                                                  +
 SUMMARY:ical generated event\r                                +
 END:VEVENT\r                                                  +
 BEGIN:X-CALENDARSERVER-PERUSER\r                              +
 UID:CDA73304-180F-48A7-9078-643993E715E5\r                    +
 X-CALENDARSERVER-PERUSER-UID:andre\r                          +
 BEGIN:X-CALENDARSERVER-PERINSTANCE\r                          +
 TRANSP:OPAQUE\r                                               +
 END:X-CALENDARSERVER-PERINSTANCE\r                            +
 END:X-CALENDARSERVER-PERUSER\r                                +
 END:VCALENDAR\r                                               +

 BEGIN:VCALENDAR\r                                             +
 VERSION:2.0\r                                                 +
 CALSCALE:GREGORIAN\r                                          +
 PRODID:-//Apple Inc.//iCal 3.0m//EN\r                         +
 BEGIN:VTIMEZONE\r                                             +
 TZID:America/New_York\r                                       +
 BEGIN:STANDARD\r                                              +
 DTSTART:20071104T020000\r                                     +
 RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU\r                      +
 TZNAME:EST\r                                                  +
 TZOFFSETFROM:-0400\r                                          +
 TZOFFSETTO:-0500\r                                            +
 END:STANDARD\r                                                +
 BEGIN:DAYLIGHT\r                                              +
 DTSTART:20070311T020000\r                                     +
 RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU\r                       +
 TZNAME:EDT\r                                                  +
 TZOFFSETFROM:-0500\r                                          +
 TZOFFSETTO:-0400\r                                            +
 END:DAYLIGHT\r                                                +
 END:VTIMEZONE\r                                               +
 BEGIN:VEVENT\r                                                +
 UID:78C21AD7-EF28-4FEB-B515-92CCB26D8E71\r                    +
 DTSTART;TZID=America/New_York:20110214T160000\r               +
 DTEND;TZID=America/New_York:20110214T170000\r                 +
 DESCRIPTION:First line\nSecond line\nThird line added later.\r+
 DTSTAMP:20110214T201826Z\r                                    +
 LAST-MODIFIED:20110214T201826Z\r                              +
 SEQUENCE:0\r                                                  +
 SUMMARY:iOS generated event\r                                 +
 END:VEVENT\r                                                  +
 BEGIN:X-CALENDARSERVER-PERUSER\r                              +
 UID:78C21AD7-EF28-4FEB-B515-92CCB26D8E71\r                    +
 X-CALENDARSERVER-PERUSER-UID:andre\r                          +
 BEGIN:X-CALENDARSERVER-PERINSTANCE\r                          +
 TRANSP:OPAQUE\r                                               +
 END:X-CALENDARSERVER-PERINSTANCE\r                            +
 END:X-CALENDARSERVER-PERUSER\r                                +
 END:VCALENDAR\r                                               +

(2 rows)

Also, I just learned that this exact issue was encountered at CalConnect last week with an Android client. Turned out they were not handling "\n" properly on their side. They were expecting an uppercase N, not lowercase.

-dre

macosforgebot commented 13 years ago

@dreness originally submitted this as comment:8:⁠ticket:414


The server is behaving properly here so we'll close this ticket; please file a bug with any clients that don't treat \n as line break.

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:9:⁠ticket:414


as written in the comments above I have -- different than you -- \\n in the database.

Therefore the clients show \n this is expected.

Please remember that the mail which is sent out as invitation still is correct.

macosforgebot commented 13 years ago

@dreness originally submitted this as comment:10:⁠ticket:414


Replying to mail@…:

as written in the comments above I have -- different than you -- \\n in the database.

Therefore the clients show \n this is expected.

Please remember that the mail which is sent out as invitation still is correct.

Does this problem reproduce on your end when using only iOS or iCal clients?

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:11:⁠ticket:414


Yes it does. For more information ave a look at comment:4.

macosforgebot commented 13 years ago

@dreness originally submitted this as comment:12:⁠ticket:414


Replying to mail@…:

Yes it does. For more information ave a look at comment:4.

Can you please check the encoding of your caldav database? You can use the psql --list command for this, e.g.:

part5:~ admin$ psql -U admin caldav -l
                            List of databases
    Name    | Owner  | Encoding  | Collation | Ctype | Access privileges 
------------+--------+-----------+-----------+-------+-------------------
 augments   | admin  | SQL_ASCII | C         | C     | 
 caldav     | caldav | SQL_ASCII | C         | C     | 
 phppgadmin | admin  | SQL_ASCII | C         | C     | 
 postgres   | admin  | SQL_ASCII | C         | C     | 
 proxies    | admin  | SQL_ASCII | C         | C     | 
 template0  | admin  | SQL_ASCII | C         | C     | =c/admin         +
            |        |           |           |       | admin=CTc/admin
 template1  | admin  | SQL_ASCII | C         | C     | =c/admin         +
            |        |           |           |       | admin=CTc/admin
(7 rows)

We're looking for SQL_ASCII.

Thx, -dre

macosforgebot commented 13 years ago

mail@… originally submitted this as comment:13:⁠ticket:414


I have the following in my database:

fm@thinkpad:~/rpmbuild # LANG=en_US psql -U postgres caldav -l
                                   List of databases
   Name    |    Owner    | Encoding |  Collation  |    Ctype    |   Access privileges   
-----------+-------------+----------+-------------+-------------+-----------------------
 caldav    | caldav      | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 | 

But after upgrading my system PyGreSQL vom 3.8.1 to 4.0.0 it seems to work, so for me this issue is currently solved.

macosforgebot commented 13 years ago

@wsanchez originally submitted this as comment:14:⁠ticket:414