Emails coded on ISO-8859 get truncated

[molul]

Here's the email source code and a picture of how it looks in my osTicket installation:

MIME-Version: 1.0 Received: by 10.58.151.6 with HTTP; Thu, 27 Feb 2014 11:34:27 -0800 (PST) Date: Thu, 27 Feb 2014 20:34:27 +0100 Delivered-To: luismorcillo@gmail.com Message-ID: CAPUXZb7257DPQmZA25qudD93P7+j2=Xh2ePGW02vtdP6fKXxqw@mail.gmail.com Subject: =?ISO-8859-1?Q?Ay=FAdame?= From: Luis Morcillo luismorcillo@gmail.com To: support18 support18@brainstorm.es Content-Type: multipart/alternative; boundary=047d7bdc8c1acaccb204f368681a

--047d7bdc8c1acaccb204f368681a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

This is an ISO-8859 coded test email.

The title has a spanish word with accent, and due to that, the ticket subject will be renamed to "[No subject]".

On the other side, the body will get truncated (not blank, just truncated) if I post a word with tilde. Here it is: V=E1monos!

--=20 Luis Morcillo Luque

--047d7bdc8c1acaccb204f368681a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">

This is an ISO-8859 coded test email. The title has a= spanish word with accent, and due to that, the ticket subject will be rena= med to "[No subject]".

On the other = side, the body will get truncated (not blank, just truncated) if I post a w= ord with tilde. Here it is: V=E1monos!

--
Luis Morcillo Luque

[greezybacon]

Would you submit the md5 hash of include/htmLawed.php

Am Feb 28, 2014 um 15:04 schrieb molul notifications@github.com:

Here's the email source code and a picture of how it looks in my osTicket installation:

MIME-Version: 1.0 Received: by 10.58.151.6 with HTTP; Thu, 27 Feb 2014 11:34:27 -0800 (PST) Date: Thu, 27 Feb 2014 20:34:27 +0100 Delivered-To: luismorcillo@gmail.com Message-ID: Subject: =?ISO-8859-1?Q?Ay=FAdame?= From: Luis Morcillo luismorcillo@gmail.com To: support18 support18@brainstorm.es Content-Type: multipart/alternative; boundary=047d7bdc8c1acaccb204f368681a

--047d7bdc8c1acaccb204f368681a Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

This is an ISO-8859 coded test email.

The title has a spanish word with accent, and due to that, the ticket subject will be renamed to "[No subject]".

On the other side, the body will get truncated (not blank, just truncated) if I post a word with tilde. Here it is: V=E1monos!

--=20 Luis Morcillo Luque

--047d7bdc8c1acaccb204f368681a Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable

This is an ISO-8859 coded test email. The title has a= spanish word with accent, and due to that, the ticket subject will be rena= med to "[No subject]".

On the other = side, the body will get truncated (not blank, just truncated) if I post a w= ord with tilde. Here it is: V=E1monos!

Luis Morcillo Luque --047d7bdc8c1acaccb204f368681a--

Thanks in advance! :)

\ Reply to this email directly or view it on GitHub.

[molul]

Hi Jared. Where would that hash be? I've opened htmlLawed.php but found nothing with "md5" or "hash" keywords. Sorry, it's monday morning -_-

[Hamsteren]

You could depending on your os do (for instance in Ubuntu terminal): md5sum htmLawed.php Which for me gives (i'm not on 1.8.1) 8b893e6bf244e2a25859df0deede7397 htmLawed.php

[molul]

Oh ok, I got it: MD5 (htmLawed.php) = 8b893e6bf244e2a25859df0deede7397

That's funny. Then I'm not on 1.8.1 too? But I updated last week! :)

[Hamsteren]

It's monday no doubt that is from a 1.8.1 (forgot i installed it to test) ;) So yes that looks like the correct one for 1.8.1

[molul]

Good. That's a start! :) I'm currently doing some css adjustments to my osTicket, but I'd like to try to fix this myself later. I'll tell you if I find anything useful.

[greezybacon]

And from the system information page (Admin Panel -> Dashboard -> Information), do you have the mbstring module installed?

[molul]

Yes, its checkbox is checked.

[molul]

I've trying to track this problem. This part of the code is the key:

    if (!$charset || in_array(strtolower(trim($charset)),
            array('default','x-user-defined','iso','us-ascii')))
        $charset = 'ISO-8859-1';

    $original = $text;
    if (function_exists('iconv'))
        $text = iconv($charset, $encoding.'//IGNORE', $text);
    elseif (function_exists('mb_convert_encoding'))
        $text = mb_convert_encoding($text, $encoding, $charset);
    elseif (!strcasecmp($encoding, 'utf-8')
            && function_exists('utf8_encode')
            && !strcasecmp($charset, 'ISO-8859-1'))
        $text = utf8_encode($text);

    // If $text is false, then we have a (likely) invalid charset, use
    // the original text and assume 8-bit (latin-1 / iso-8859-1)
    // encoding
    return (!$text && $original) ? $original : $text;

More precisely, here:

    if (function_exists('iconv'))
        $text = iconv($charset, $encoding.'//IGNORE', $text);

I printed $original and $text variables inside that if. $original is fine, and $text is empty. I'm gonna check why.

---

EDIT: the problem is with the "//IGNORE" flag. Here's an explanation: http://www.php.net/manual/en/function.iconv.php#108643

---

EDIT2: in include/class.format.php, if I change this: if (function_exists('iconv')) { $text = iconv($charset, $encoding.'//IGNORE', $text); } elseif (function_exists('mb_convert_encoding')) {

for this: / if (function_exists('iconv')) { $text = iconv($charset, $encoding.'//IGNORE', $text); } elseif (function_exists('mb_convert_encoding')) { / if (function_exists('mb_convert_encoding')) {

Then the ticket body comes perfect, with tildes. Still get a "[No subject]". Gonna take a look at that.

Is using "iconv" function something critical?

---

EDIT3: if I edit include/pear/Mail/mimeDecode.php changing this:

        if($charset && $this->_charset && strcasecmp($this->_charset, $charset)) {
            if(function_exists('iconv'))
                $text = iconv($charset, $this->_charset.'//IGNORE', $text);
            elseif(function_exists('mb_convert_encoding'))

for this:

        if($charset && $this->_charset && strcasecmp($this->_charset, $charset)) {

/ if(function_exists('iconv')) $text = iconv($charset, $this->_charset.'//IGNORE', $text); elseif(function_exists('mb_convert_encoding')) / if(function_exists('mb_convert_encoding'))

I get my subject with tildes and all the stuff. Tested it with utf8 and iso-8859. All tickets are created perfectly. I think I'll go with this workaround for now, as nothing seems broken (we'll see) :)

[greezybacon]

Could you submit the LC_* environment variables from your server? (Perhaps from a phpinfo() call)

[molul]

Hmmm... I did that phpinfo(), then search for something with "LC_", but no search results :(

[molul]

Ouch! Today I made my company's new osTicket (1.8.1) system publicly available, and some russian customers say they receive empty emails. I guess it's related to this. I'll investigate and get back to you.

[greezybacon]

There was an issue with the html2text that could corrupt text blocks with non-breaking spaces in them

[simonkurka]

Hi,

I have the same problem and can trace the error to the exact same function. Is there any good reason why osTicket should rely on a PHP function which has a known bug since a long time?

May I start a Pull-Request for the suggestion of @molul ? I tested it and it works (latest Debian Wheezy packages)!