search

roundcube / roundcubemail

The Roundcube Webmail suite
https://roundcube.net
GNU General Public License v3.0
5.79k stars 1.62k forks source link

Issue with umlauts within an email attachment #7949

Closed uli-heller closed 3 years ago

uli-heller commented 3 years ago

Maybe the same issue as in #7139 ?

I'm using roundcube-1.4.11. I do have a linux desktop.

Here is what I do:

  1. Create a file: /tmp/uli.txt - äöüÄÖÜß
  2. Show the content of the file: 
    $ cat /tmp/uli.txt
äöüÄÖÜß
$ od -cx /tmp/uli.txt
0000000 303 244 303 266 303 274 303 204 303 226 303 234 303 237  \n
           a4c3    b6c3    bcc3    84c3    96c3    9cc3    9fc3    000a
0000017
  3. Send an email to myself, add the file as an attachment
  4. Receive the email, save the attachment and look at the file - exactly as before
  5. Look at the attachment within roundcube: umlauts are are garbled

image

alecpl commented 3 years ago

Works for me. Enable imap_debug and check the log.

uli-heller commented 3 years ago

Done that. Which part of the log is relevant?

When trying to show the content of the attachment, something like this shows up:

[18-Mar-2021 07:48:10 +0100]: [AC14] S: Content-Transfer-Encoding: base64 Content-Type: text/plain; name=uli.txt Content-Disposition: attachment; filename=uli.txt; size=15

Best regards, Uli

​

Am 2021-03-18 07:18, schrieb Aleksander Machniak:

Works for me. Enable imap_debug and check the log.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801662257 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZXWE7V6NNBPFLOTCWLTEGLM3ANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

The relevant part will be the body and bodystructure (with surrounding text).

uli-heller commented 3 years ago

Is it this:

[18-Mar-2021 07:48:10 +0100]: [AC14] C: A0007 UID FETCH 12707 (BODY.PEEK[2.MIME]) [18-Mar-2021 07:48:10 +0100]: [AC14] S: * 7806 FETCH (UID 12707 BODY[2.MIME] {143} [18-Mar-2021 07:48:10 +0100]: [AC14] S: Content-Transfer-Encoding: base64 Content-Type: text/plain; name=uli.txt Content-Disposition: attachment; filename=uli.txt; size=15

[18-Mar-2021 07:48:10 +0100]: [AC14] S: ) [18-Mar-2021 07:48:10 +0100]: [AC14] S: A0007 OK Fetch completed (0.016 + 0.000 + 0.015 secs). [18-Mar-2021 07:48:10 +0100]: [AC14] C: A0008 UID FETCH 12707 (BINARY.PEEK[2]) [18-Mar-2021 07:48:10 +0100]: [AC14] S: 7806 FETCH (UID 12707 BINARY[2] ~{15} [18-Mar-2021 07:48:10 +0100]: [AC14] S: äöüÄÖÜß [18-Mar-2021 07:48:10 +0100]: [AC14] S: ) [18-Mar-2021 07:48:10 +0100]: [AC14] S: A0008 OK Fetch completed (0.001 + 0.000 secs). [18-Mar-2021 07:48:10 +0100]: [AC14] C: A0009 LOGOUT [18-Mar-2021 07:48:10 +0100]: [AC14] S: BYE Logging out [18-Mar-2021 07:48:10 +0100]: [AC14] S: A0009 OK Logout completed (0.001 + 0.000 secs).

I see the content of the attachment. It looks good.

Please note: I'm using chrome browser typically. Firefox shows the same behavior, so the umlauts don't show up correctly for chromne and firefox.

​

Am 2021-03-18 07:59, schrieb Aleksander Machniak:

The relevant part will be the body and bodystructure (with surrounding text).

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801681422 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZSHRM7XCXNGHNT7M4LTEGQEFANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

Looks good. I would check the raw content of the http response with the attachment body in the web browser, including headers. So we see what exactly is sent by Roundcube to the browser, the body might be wrong or the charset in headers is somehow wrong.

uli-heller commented 3 years ago

when downloading, the umlauts are fine. The content type looks fine there.

For diaplay, UTF-8 seems to be missing.

How can I look at the response of the display part? It opens a new window without devtools unfortunately.

Am 2021-03-18 08:12, schrieb Aleksander Machniak:

Looks good. I would check the raw content of the http response with the attachment body in the web browser, including headers. So we see what exactly is sent by Roundcube to the browser, the body might be wrong or the charset in headers is somehow wrong.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

image

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801689027 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZSM2LSEO4NTLP7QE6DTEGRW5ANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

Ctrl+Shift+I to open the console and refresh the page.

uli-heller commented 3 years ago

Thx, here is the screenshot.

Content-Type looks wrong to me.

Clicking on "Reponse" shows wrong umlauts

Am 2021-03-18 08:40, schrieb Aleksander Machniak:

Ctrl+Shift+I to open the console and refresh the page.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801703615 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZSDFG4IJB3MCPPOL4LTEGU6XANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

I don't see the screenshot. You have to attach it to the ticket.

uli-heller commented 3 years ago

image

uli-heller commented 3 years ago

image

uli-heller commented 3 years ago

Sorry. I added both screenshots. The the first one is the latest created by your instructions. The 2nd one compares display with download.

Am 2021-03-18 09:20, schrieb Aleksander Machniak:

I don't see the screenshot. You have to attach it to the ticket.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801726307 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZVBNZDQN6GNQWZCDPTTEGZWBANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

Go back to the imap debug log and find the BODYSTRUCTURE response.

uli-heller commented 3 years ago

TRANSFER-ENCODING REFERENCES X-DRAFT-INFO MAIL-FOLLOWUP-TO MAIL-REPLY-TO RETURN-PATH)]) [18-Mar-2021 07:46:51 +0100]: [88A3] S: * 7806 FETCH (UID 12707 RFC822.SIZE 1619 FLAGS (\Seen) INTERNALDATE "16-Mar-2021 15:07:01 +0100" BODYSTRUCTURE (("text" "plain" ("charset" "UTF-8" "format" "flowed") NIL NIL "8bit" 380 12 NIL NIL NIL NIL)("text" "plain" ("name" "uli.txt" "charset" "us-ascii") NIL NIL "base64" 20 0 NIL ("attachment" ("filename" "uli.txt" "size" "15")) NIL NIL) "mixed" ("boundary" "=_00566522bf5bbd76fc02d201167bb230") NIL NIL NIL) BODY[HEADER.FIELDS (DATE FROM TO SUBJECT CONTENT-TYPE CC REPLY-TO LIST-POST DISPOSITION-NOTIFICATION-TO X-PRIORITY X-KOLAB-TYPE X-KOLAB-MIME-VERSION MESSAGE-ID IN-REPLY-TO BCC SENDER CONTENT-TRANSFER-ENCODING REFERENCES X-DRAFT-INFO MAIL-FOLLOWUP-TO MAIL-REPLY-TO RETURN-PATH)] {376} [18-Mar-2021 07:46:51 +0100]: [88A3] S: Return-Path: @.> Content-Type: multipart/mixed; boundary="=_00566522bf5bbd76fc02d201167bb230" Content-Transfer-Encoding: 8bit Date: Tue, 16 Mar 2021 15:07:01 +0100 From: Uli Heller @.> To: Uli Heller @.> Subject: Test Message-ID: @.>

[18-Mar-2021 07:46:51 +0100]: [88A3] S: ) [18-Mar-2021 07:46:51 +0100]: [88A3] S: A0006 OK Fetch completed (0.008 + 0.000 + 0.007 secs). [18-Mar-2021 07:46:51 +0100]: [88A3] C: A0007 UID FETCH 12707 (BODY.PEEK[2.MIME]) [18-Mar-2021 07:46:51 +0100]: [88A3] S: * 7806 FETCH (UID 12707 BODY[2.MIME] {143} [18-Mar-2021 07:46:51 +0100]: [88A3] S: Content-Transfer-Encoding: base64 Content-Type: text/plain; name=uli.txt Content-Disposition: attachment; filename=uli.txt; size=15

[18-Mar-2021 07:46:51 +0100]: [88A3] S: ) [18-Mar-2021 07:46:51 +0100]: [88A3] S: A0007 OK Fetch completed (0.013 + 0.000 + 0.012 secs). [18-Mar-2021 07:46:51 +0100]: [88A3] C: A0008 LIST (SUBSCRIBED) "" "*"

...

[18-Mar-2021 07:48:10 +0100]: [3B8F] C: A0006 UID FETCH 12707 (UID RFC822.SIZE FLAGS INTERNALDATE BODYSTRUCTURE BODY.PEEK[HEADER.FIELDS (DATE FROM TO SUBJECT CONTENT-TYPE CC REPLY-TO LIST-POST DISPOSITION-NOTIFICATION-TO X-PRIORITY X-KOLAB-TYPE X-KOLAB-MIME-VERSION MESSAGE-ID IN-REPLY-TO BCC SENDER CONTENT-TRANSFER-ENCODING REFERENCES X-DRAFT-INFO MAIL-FOLLOWUP-TO MAIL-REPLY-TO RETURN-PATH)]) [18-Mar-2021 07:48:10 +0100]: [3B8F] S: * 7806 FETCH (UID 12707 RFC822.SIZE 1619 FLAGS (\Seen) INTERNALDATE "16-Mar-2021 15:07:01 +0100" BODYSTRUCTURE (("text" "plain" ("charset" "UTF-8" "format" "flowed") NIL NIL "8bit" 380 12 NIL NIL NIL NIL)("text" "plain" ("name" "uli.txt" "charset" "us-ascii") NIL NIL "base64" 20 0 NIL ("attachment" ("filename" "uli.txt" "size" "15")) NIL NIL) "mixed" ("boundary" "=_00566522bf5bbd76fc02d201167bb230") NIL NIL NIL) BODY[HEADER.FIELDS (DATE FROM TO SUBJECT CONTENT-TYPE CC REPLY-TO LIST-POST DISPOSITION-NOTIFICATION-TO X-PRIORITY X-KOLAB-TYPE X-KOLAB-MIME-VERSION MESSAGE-ID IN-REPLY-TO BCC SENDER CONTENT-TRANSFER-ENCODING REFERENCES X-DRAFT-INFO MAIL-FOLLOWUP-TO MAIL-REPLY-TO RETURN-PATH)] {376} [18-Mar-2021 07:48:10 +0100]: [3B8F] S: Return-Path: @.> Content-Type: multipart/mixed; boundary="=_00566522bf5bbd76fc02d201167bb230" Content-Transfer-Encoding: 8bit Date: Tue, 16 Mar 2021 15:07:01 +0100 From: Uli Heller @.> To: Uli Heller @.> Subject: Test Message-ID: @.>

[18-Mar-2021 07:48:10 +0100]: [3B8F] S: ) [18-Mar-2021 07:48:10 +0100]: [3B8F] S: A0006 OK Fetch completed (0.006 + 0.000 + 0.005 secs). [18-Mar-2021 07:48:10 +0100]: [3B8F] C: A0007 UID FETCH 12707 (BODY.PEEK[2.MIME]) [18-Mar-2021 07:48:10 +0100]: [3B8F] S: * 7806 FETCH (UID 12707 BODY[2.MIME] {143} [18-Mar-2021 07:48:10 +0100]: [3B8F] S: Content-Transfer-Encoding: base64 Content-Type: text/plain; name=uli.txt Content-Disposition: attachment; filename=uli.txt; size=15

[18-Mar-2021 07:48:10 +0100]: [3B8F] S: ) [18-Mar-2021 07:48:10 +0100]: [3B8F] S: A0007 OK Fetch completed (0.013 + 0.000 + 0.012 secs).

...

​

Am 2021-03-18 09:35, schrieb Aleksander Machniak:

Go back to the imap debug log and find the BODYSTRUCTURE response.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801735023 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZT3EJIT45SW6W6C3TDTEG3NFANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

("text" "plain" ("name" "uli.txt" "charset" "us-ascii")

This part in the response is wrong. It says the attachment is in this charset. So, this is an imap server bug. What imap server do you use?

I guess we could workaround this case by using the default UTF-8 charset when the imap server says us-ascii.

uli-heller commented 3 years ago

I'm using dovecot. I'll look at the documentation, maybe I can fix it there. I'll be back, maybe tomorrow... Thx a lot!

Am 2021-03-18 09:55, schrieb Aleksander Machniak:

("text" "plain" ("name" "uli.txt" "charset" "us-ascii")

This part in the response is wrong. It says the attachment is in this charset. So, this is an imap server bug. What imap server do you use?

I guess we could workaround this case by using the default UTF-8 charset when the imap server says us-ascii.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801747740 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZXA5EQBUWLIEQJKPD3TEG5XRANCNFSM4ZIUNWAQ

uli-heller commented 3 years ago

I think I'm going to do something different, hope this is fine with you:

* sending and receiving the attachment via roundcube shows the issue
* sending via zammad and receiving via roundcube doesn't show the

issue

Maybe it is a good idea to look at the imap log for both of these? Maybe we (or better: you) can fix/change the roundcube sending process?

Does this sound reasonable to you?

Best regards, Uli

​

Am 2021-03-18 09:55, schrieb Aleksander Machniak:

("text" "plain" ("name" "uli.txt" "charset" "us-ascii")

This part in the response is wrong. It says the attachment is in this charset. So, this is an imap server bug. What imap server do you use?

I guess we could workaround this case by using the default UTF-8 charset when the imap server says us-ascii.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801747740 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZXA5EQBUWLIEQJKPD3TEG5XRANCNFSM4ZIUNWAQ

alecpl commented 3 years ago

The other client probably sets the charset in the attachment part headers, so the imap server does not fallback to us-ascii. Making sure Roundcube sets the charset for attachments in sent mail would probably workaround this issue too, but might be not that easy. Detecting charset does not always work reliably.

uli-heller commented 3 years ago

Thx for the feedback, you're probably right.

I checked dovecot but there is probably no easy way to change the default charset there. So maybe I have to live with the issue.

Best regards, Uli

Am 2021-03-18 15:50, schrieb Aleksander Machniak:

The other client probably sets the charset in the attachment part headers, so the imap server does not fallback to us-ascii. Making sure Roundcube sets the charset for attachments in sent mail would probably workaround this issue too, but might be not that easy. Detecting charset does not always work reliably.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801991136 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZWY6ARFF56M4WDHTLDTEIHL3ANCNFSM4ZIUNWAQ

uli-heller commented 3 years ago

I'm done with the tests, unfortunately the results are as expexted.

    SENDER
    “DISPLAY” WITHIN ROUNDCUBE

    roundcube-1.4.11
    KO, umlauts look strange within roundcube

    zammad-3.6
    OK, umlauts are fine within roundcube

    thunderbird
    OK, umlauts are fine within roundcube

So there is an issue when sending and receiving attachments containing umlauts when using roundcube on both ends.

​

Am 2021-03-18 15:50, schrieb Aleksander Machniak:

The other client probably sets the charset in the attachment part headers, so the imap server does not fallback to us-ascii. Making sure Roundcube sets the charset for attachments in sent mail would probably workaround this issue too, but might be not that easy. Detecting charset does not always work reliably.

-- You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub [1], or unsubscribe [2].

Links:

[1] https://github.com/roundcube/roundcubemail/issues/7949#issuecomment-801991136 [2] https://github.com/notifications/unsubscribe-auth/AAMZQZWY6ARFF56M4WDHTLDTEIHL3ANCNFSM4ZIUNWAQ

jsoref commented 3 years ago

Fwiw, https://www.w3.org/Protocols/rfc1341/7_1_Text.html:

7.1 The Text Content-Type

The text Content-Type is intended for sending material which is principally textual in form. It is the default Content-Type. A "charset" parameter may be used to indicate the character set of the body text. The primary subtype of text is "plain". This indicates plain (unformatted) text. The default Content-Type for Internet mail is "text/plain; charset=us-ascii".

If someone wants to send something that isn't us-ascii, it's really their obligation to appropriately tag it.

Speaking as someone who has worked on browsers, email clients, web standards, and occasionally heckles servers.

alecpl commented 3 years ago

Here's more recent RFC https://tools.ietf.org/html/rfc2045#section-5.2

To sum this up. I see two issues here:

  1. Roundcube does not set the attachment charset. As I said before this is not an easy task. Also, I don't think this is a bug to not include the charset.
  2. The imap server says (in the BODYSTRUCTURE response) the charset is us-ascii even though the header does not specify the charset. This in my opinion is a bug.

That being said. It is a wontfix. Feel free to create a feature request to detect encoding of plain text attachments, but I'm afraid this might be hard to do properly.

uli-heller commented 3 years ago

I created 2 feature requests: #8008 and #8009 . I do understand that the "real" error is not within roundcube. But on the other hand, it is not very good for roundcube's reputation when roundcube-to-roundcube communication fails (with the probably most common IMAP server dovecot in between) but thunderbird-to-roundcube works fine. Please don't feel offended by this statement, just my personal feeling. I really appreciate the hard work you're putting into roundcube. I really like it. It is a very good mailclient.