broken preview for HTML messages on iOS

[ramcq]

I'm using Z-Push-contrib up to date with 49bc9a3 and have just added a new user with an iPhone. He's told me that HTML messages are previewing incorrectly - as the screenshot shows, the HTML source is provided as a preview, rather than the text/plain part.

An example message source (the 2nd in this screenshot) is at http://ramcq.net/files/tmp/Z-Push-HTML-email-example.eml (for the time being). Let me know how/if I can help debug.

[fmbiete]

Referenced commit should fix the problem. Thanks!!!!

[chinenual]

Fixes the problem for me! Thanks.

[fmbiete]

Fixed then, thanks!!!

[chinenual]

I commented earlier that the fix worked - and it does seem to work for at least one class of emails I used to see w/ HTML notifications. But I'm still seeing raw-HTML notifications - probably should reopen this issue. How can I help debug?

[yodax]

@fmbiete I see this issue as well.

Some previews start with <!DOCTYPE some with <p. I think it is either partial html or it has a doc type. If I can help debugging let me know.

[yodax]

@fmbiete I did some more digging. I don't think iOS is requesting a preview.

On this line: if (Request::GetProtocolVersion() >= 14.0 && $bpo->GetPreview()) {

Protocol version is 14 and GetPreview is 0, therefore the code isn't triggered. Why however I do not know.

[fmbiete]

I think we could be having 2 different issues here:

• Maybe the message doesn't have plain/text part. The code tries to convert html tags to create one, but it only remove basic tags. If this is the problem we should use a library to strip all of them. @chinenual Can you check the raw messages where you are seeing a wrong preview? Even sending me an example would be great.
• Maybe the device is not asking for preview but trying to show it. This would be a client bug. But I think we could always fill the preview variable (remove the $bpo->GetPreview() in that condition). @yodax Can you check if that solve your case?

[yodax]

I tried that, didn't work. I also tried assigning a static string in the preview field. That wasn't picked up on the device.

Weird thing is, that with an old version of zpush it did work. I ran a diff between this commit (that mail in a box used to use) and the latest but I couldn't find any obvious regressions.

That suggests that it might not be a problem with imap.php but somewhere else. Problem is I don't know enough about this library to know how everything ties together. With regards to the backend, states and frontend.

[chinenual]

I've emailed one of the email messages that triggers the bug for me. Preview shows raw HTML, even though when I click through to view the full message text, the message is properly decoded.

I've redacted some of the headers (e.g. received, etc.) but more than likely the problem has something to do with the mime encoding. Hope this helps track down the problem.

[yodax]

@chinenual in your case do you also notice that after a full resync the messages in iOS appear first and the sequentially the previews? And that they are regenerated after a while.

[m-emelchenkov]

The same for me. Original Z-Push works OK. Too much differences in imap.php so can't identify what's wrong.

[basbebe]

Same problem here

[basbebe]

I also get a lot of these logs:

28/12/2015 19:11:37 [63884] [DEBUG] [XXX@YYYZZZ.com] [ABCDEFGxxxxx] BackendIMAP->GetMessage(): no extension found in '/usr/local/etc/nginx/mime.types'!!

[fmbiete]

@bax- That last message is telling you that you don't have correctly configured the path to the mime.types file. That file will help to determine the file type for attachments.

[fmbiete]

@chinenual The message you sent me looks ok, but it has a quoted part, it's not really plain.

I'm starting to suspect that your phone is not requesting the preview but trying to make one on the fly...

Can you try commenting these lines in imap.php? 1146: if (Request::GetProtocolVersion() >= 14.0 && $bpo->GetPreview()) { 1155: }

This will force to fill the preview always, even if the device is not asking for it.

[jav4]

This a multipart mail, with text/plain and text/html. Log for preview:

BackendIMAP->StatMessage('6218ab78','53') BackendIMAP->getImapIdFromFolderId('6218ab78') = INBOX BackendIMAP->GetMessage('6218ab78', '53', '1') BackendIMAP->getImapIdFromFolderId('6218ab78') = INBOX BackendIMAP->StatMessage('6218ab78','53') BackendIMAP->getImapIdFromFolderId('6218ab78') = INBOX BackendIMAP->GetMessage(): Message is multipart: 1, smime: 0, smime encrypted: 0 BackendIMAP->GetMessage(): getBodyPreferenceBestMatch: 1 BackendIMAP->GetMessage(): after thinking a bit we will use: 2

So it sends text/html to the iPhone, when it's requesting text/plain. I don't understand why it decides to send html anyway.

[basbebe]

@fmbiete /usr/local/etc/nginx/mime.typesis the correct location of my mime.types file. Should I add something to it? The permissions are: -rw-r--r--

The previews stay broken (HTML tags in the iOS list)

[fmbiete]

@bax- You have to change your imap/config.php

// A file containing file mime types->extension mappings.
//  SELINUX users: make sure the file has a security context accesible by your apache/php-fpm process
define('SYSTEM_MIME_TYPES_MAPPING', '/etc/mime.types');

Let's check if the device is really using the returned preview. Replace this

                if (Request::GetProtocolVersion() >= 14.0 && $bpo->GetPreview()) {
                    // Preview must be always plaintext
                    $previewText = "";
                    Mail_mimeDecode::getBodyRecursive($message, "plain", $previewText, true);
                    if (strlen($previewText) == 0) {
                        Mail_mimeDecode::getBodyRecursive($message, "html", $previewText, true);
                        $previewText = Utils::ConvertHtmlToText($previewText);
                    }
                    $output->asbody->preview = Utils::Utf8_truncate($previewText, $bpo->GetPreview());
                }

by this

$output->asbody->preview = "FAKE PREVIEW";

After a full resync you only should see "FAKE PREVIEW", if you see something else...

@jav4 Increase your log level to debug at least. BUT, what you are looking is the body message. Here we are speaking about the preview.

[jav4]

@jav4 https://github.com/jav4 Increase your log level to debug at least. BUT, what you are looking is the body message. Here we are speaking about the preview.

This comes from a "preview" request from an iPhone. It asks for a text/plain body to be shown as preview, but this isn't used for full message presentation. Full body is downloaded as html for presentation.

[basbebe]

@fmbiete I made the change in imap.phpbut after a resin the previews stay the same.

Here is another excerpt from the last log:

05/01/2016 10:21:12 [75911] [DEBUG] [XXX@YYYZZZ.com] BackendIMAP->GetMessage(): Guessing extension for inline attachment [primary_type message secondary_type delivery-status]
05/01/2016 10:21:12 [75911] [DEBUG] [XXX@YYYZZZ.com] BackendIMAP->GetMessage(): no extension found in '/usr/local/etc/nginx/mime.types'!!
05/01/2016 10:21:12 [75911] [DEBUG] [XXX@YYYZZZ.com] BackendIMAP->GetMessage(): Guessing extension for inline attachment [primary_type message secondary_type rfc822]
05/01/2016 10:21:12 [75911] [DEBUG] [XXX@YYYZZZ.com] BackendIMAP->GetMessage(): no extension found in '/usr/local/etc/nginx/mime.types'!!

maybe there are types that I should add to my mime-types file?

[fmbiete]

@jav4 No, sorry. We don't have any debug message indicating the type of body for the preview, so you won't see anything related to it in the logs. All those messages are for the body.

@bax- Can you double check that you don't see "FAKE PREVIEW"? If you don't see that text, it means that your device is ignoring the text returned by z-push and creating a preview from the body... and we couldn't do anything about it...

[basbebe]

I checked again: I did the change in map.php, restarted php-fpm, forced a resync. But the previews stay the same: HTML source. I remember that an older version of Z-Push didn't have that problem.

[fmbiete]

I don't see how we can fix that if the phone doesn't use the preview z-push is sending. Sorry.

[tboerner]

After a couple of months I'd like to reopen this issue, as it is really bugging me. Basically I can confirm everything mentioned above. Some previews / notifications are displayed correctly, others are not. Apparently it works for multi-part message in MIME format, but not for simple plain/html messages. For testing purpose, I've switched temporarily to original Z-Push v.2.2.9, and I made the following observations: Broken previews are still broken but can be fixed by moving the message to a different folder (and optionally back again). Obviously then the preview is rebuilt, and shows up correctly afterwards. However, in my case after the version change for new messages both preview and body were displayed in raw HTML format. I could fix this by resyncing the device. After the resync all old and new HTML messages are displayed correctly with plain preview and HTML body.

Conclusion: The issue can be solved on server side.

Another observation, now back on SVN-trunk-r1981: I remember that once I've watched a HTML message loading on the iPhone, showing the preview correctly for less than half a second and then presenting the complained HTML code. Could it be that the preview is loaded twice, first correctly as plain text but then overwritten by the HTML code? This would explain why the "FAKE PREVIEW" wasn't displayed in bax's test. Maybe it actually was displayed, but just too short for being seen.

Please let me know how I can contribute to fix this behavior. I have iDevices available for testing. The mail server runs on Debian jessie with nginx 1.6.1.

[emcepe]

Hello,

I have the same problems described here. Preview of HTML E-Mails is always raw html, even completely new installed devices. I'm using z-push since 3 days... If connected to a real M$ Exchange, it works as expected.

[ghost]

Can confirm I'm seeing the same issue on a fresh install of iOS 10.

[Mikka20]

Can confirm this issue too. I'm seeing the same on a fresh install of iOS 10 devices.

[dformdotdk]

Same problem here on iOS 10 with latest nightly build ( version 2.4 )

[m4teh]

This is still happening. iOS 11.

[tboerner]

Same here.

A time ago, I experimentally replaced the IMAP backend with the one from fmbiete's fork. This fixed the issue, but it is not a solution, as it requires lot of adaptations in other parts of the project - those two branches are meanwhile too different from each other. The two IMAP backends are so different from each other that I couldn't see at which place the bug may have been introduced.

However, it shows that in principle a fix exists. Perhaps one of the original authors can have a look at it and will hopefully see which change can be the cause of the issue.

Cheers, Thomas

Am 25.06.2018 03:32, schrieb Mathew Estrada:

This is still happening. iOS 11.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub [1], or mute the thread [2].

Links:

[1] https://github.com/fmbiete/Z-Push-contrib/issues/229#issuecomment-399806644 [2] https://github.com/notifications/unsubscribe-auth/ASfLBZQ8RuPY0DHAjC_3f9yg5YNKdYW4ks5uAD2vgaJpZM4F-TRh

[treellama]

I just upgraded from an ancient version of d-push (2.0.7-1) to z-push 2.4.4 (not z-push-contrib), and noticed this problem. I looked at the old code, and in backend/imap/imap.php, after:

if (Request::GetProtocolVersion() >= 14.0 && $bpo->GetPreview()) {
...
}

I restored this clause:

else {
    $output->asbody->truncated = 0;
}

With that change, I now see the correct previews. Perhaps that is a clue as to the cause/solution of this bug? The truncated member variable?

[emcepe]

I tried your fix but for me it is not working :-( ... I still see raw HTML preview

[treellama]

Yeah, I still see raw HTML previews sometimes even with that change. But then later the same messages display correctly. Then later later, the raw HTML comes back.

[mbroemme]

Replace the following:

if ($bpReturnType == SYNC_BODYPREFERENCE_MIME || $is_smime || in_array(SYNC_BODYPREFERENCE_MIME, $bodypreference)) {

with:

if ($bpReturnType == SYNC_BODYPREFERENCE_MIME || $is_smime || $is_multipart || in_array(SYNC_BODYPREFERENCE_MIME, $bodypreference)) {

at line 1061 in: https://github.com/fmbiete/Z-Push-contrib/blob/682f127b7e5dc77da95eef6956579215d376b0e5/backend/imap/imap.php#L1061

After that do the following z-push-admin.php -a resync -u <user> -d <device> and refresh your mail account on iPhone.

This should fix it for majority of mails. Actually multipart mails must be MIME encoded and it is save to set SYNC_BODYPREFERENCE_MIME in this case.

[pfisterer]

@mbroemme I patched Z-Push-contrib/backend/imap/imap.php as described, triggered a resync - but I still do see raw html previews.

I'm running z-push 2.4.5-2 on Debian 10 and use an iPhone running iOS 13.3.1.

Any ideas?

[sdondley]

I'm also on Debian 10 with stock debian z-push package and I am experiencing this problem. The suggestion from @mbroemme worked for most emails and they are now showing properly. There were some spam emails in my inbox that it did not work on. I'm going to assume those were not properly formatted.