search

freescout-help-desk / freescout

FreeScout — Free self-hosted help desk & shared mailbox (Zendesk / Help Scout alternative)
https://freescout.net
GNU Affero General Public License v3.0
3.09k stars 505 forks source link

Email empty #1972

Closed afawaz2 closed 1 year ago

afawaz2 commented 2 years ago

We are having an issue where external replies to conversation or new conversations that are forwarded show up as empty, when we try to show original we also get an empty page with
as the body view. The headers show up as well.

Those messages have content in them because when we go to outlook we are able to see the message.

The log does not have any parsing error or fetching errors but we are seeing this issue in 2% of the emails we receive every day. I would like to debug the issue so any pointer would help, also a workaround that doesn't involve logging to outlook every time it happens would be helpful.

afawaz2 commented 2 years ago

I just noticed a pattern of when this occurs:

  1. We send out an email and the ticket number gets appended
  2. The reply gets registered as a new conversation with the subject showing "Re: [#number] SUBJECT"
freescout-helpdesk commented 2 years ago

Do you see content via Show Original thread menu item?

afawaz2 commented 2 years ago

No image

afawaz2 commented 2 years ago

Example email header that has an empty body

MIME-Version: 1.0 Received: from CH0PR08MB7338.namprd08.prod.outlook.com (2603:10b6:610:113::17) by CH0PR08MB7369.namprd08.prod.outlook.com with HTTPS; Thu, 5 May 2022 18:26:49 +0000 Received: from MW4PR04CA0289.namprd04.prod.outlook.com (2603:10b6:303:89::24) by CH0PR08MB7338.namprd08.prod.outlook.com (2603:10b6:610:113::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.13; Thu, 5 May 2022 18:26:45 +0000 Received: from CO1NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:303:89:cafe::51) by MW4PR04CA0289.outlook.office365.com (2603:10b6:303:89::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.12 via Frontend Transport; Thu, 5 May 2022 18:26:45 +0000 Authentication-Results: spf=softfail (sender IP is XX.XX.X.XXX) smtp.mailfrom=client.com; dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=example.com;compauth=softpass reason=201 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning ersiwealth.com discourages use of XX.XX.XX.XX as permitted sender) Received: from us-smtp-delivery-127.mimecast.com (XX.XX.XX.XX) by CO1NAM11FT047.mail.protection.outlook.com (10.13.174.132) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5227.15 via Frontend Transport; Thu, 5 May 2022 18:26:44 +0000 Received: from mta5.redtailtechnology.com (mta5.redtailtechnology.com [65.74.153.222]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-518-cSxK2MBcNmSYSWakqG50mw-1; Thu, 05 May 2022 14:26:42 -0400 X-MC-Unique: cSxK2MBcNmSYSWakqG50mw-1 Received: from mta5.redtailtechnology.com (localhost [127.0.0.1]) by mta5.redtailtechnology.com (Postfix) with ESMTPS id 197F328AA66 for XXX@example.com; Thu, 5 May 2022 11:26:41 -0700 (PDT) Received: from mta5.redtailtechnology.com (localhost [127.0.0.1]) by mta5.redtailtechnology.com (Postfix) with ESMTPS id 056B928AAA0 for XXX@example.com; Thu, 5 May 2022 11:26:41 -0700 (PDT) Received: from smfeml03store005.redtailtechnology.com (unknown [192.168.5.105]) by mta5.redtailtechnology.com (Postfix) with ESMTP id DE45F28AA66 for XXX@example.com; Thu, 5 May 2022 11:26:40 -0700 (PDT) From: Jon Boejon@client.com To: XXX@example.com Subject: Person Thread-Topic: Person Date: Thu, 5 May 2022 11:26:40 -0700 (PDT) Message-ID: 025101d860ad$a943ef40$fbcbcdc0$@client.com X-Mailer: Microsoft Outlook 16.0 X-Mailer: Zimbra 9.0.0_GA_4204 (Zimbra-ZCO/8.7.10.1711 (10.0.19044 en-US) P56b4 T2d10 R2206) Thread-Index: AdhgrSy3aeaUiG6OSGqUdPLjHdtiIA== X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: client.com Content-Type: multipart/related; boundary="----=_NextPart_000_025201D8607B.5EAA4290" Content-Language: en-us Return-Path: jon@client.com X-MS-Exchange-Organization-ExpirationStartTime: 05 May 2022 18:26:45.1918 (UTC) X-MS-Exchange-Organization-ExpirationStartTimeReason: OriginalSubmit X-MS-Exchange-Organization-ExpirationInterval: 1:00:00:00.0000000 X-MS-Exchange-Organization-ExpirationIntervalReason: OriginalSubmit X-MS-Exchange-Organization-Network-Message-Id: 32d7a0c7-1e2b-4f5f-673d-08da2ec4cf40 X-EOPAttributedMessage: 0 X-EOPTenantAttributedMessage: 43c89e32-6531-4509-b389-cac81750ccb9:0 X-MS-Exchange-Organization-MessageDirectionality: Incoming X-MS-PublicTrafficType: Email X-MS-Exchange-Organization-AuthSource: CO1NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-Organization-AuthAs: Anonymous X-MS-Office365-Filtering-Correlation-Id: 32d7a0c7-1e2b-4f5f-673d-08da2ec4cf40 X-MS-TrafficTypeDiagnostic: CH0PR08MB7338:EE X-MS-Exchange-AtpMessageProperties: SA X-MS-Exchange-Organization-SCL: 1 X-Microsoft-Antispam: BCL:0; X-Forefront-Antispam-Report: CIP:170.10.133.127;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:us-smtp-delivery-127.mimecast.com;PTR:us-smtp-delivery-127.mimecast.com;CAT:NONE;SFS:(13230001)(4636009)(8676002)(6916009)(86362001)(36756003)(356005)(1096003)(166002)(58800400005)(7116003)(44736005)(966005)(22186003)(26005)(7636003)(2616005)(7596003)(6266002)(1420700001)(336012)(3480700007)(40140700001)(5660300002)(83380400001)(35090500004)(43540500003);DIR:INB; X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 18:26:44.7856 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 32d7a0c7-1e2b-4f5f-673d-08da2ec4cf40 X-MS-Exchange-CrossTenant-Id: 43c89e32-6531-4509-b389-cac81750ccb9 X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: Internet X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR08MB7338 X-MS-Exchange-Transport-EndToEndLatency: 00:00:05.0564700 X-MS-Exchange-Processed-By-BccFoldering: 15.20.5206.027 X-Microsoft-Antispam-Mailbox-Delivery: ucf:0;jmr:0;auth:0;dest:I;ENG:(910001)(944506458)(944626604)(920097)(930097); X-Microsoft-Antispam-Message-Info: =?Windows-1252?Q?CUy1jrisyROuDt2e9/DnE0QUyaS809Yh6tyrjjtB0XP55Y7eTNkAHfU5?= =?Windows-1252?Q?rZBTuGTJygO04wYf6zSLAxAbUJSLz/6av/1HIb5MgUZi7gQDjW/jdlzc?= =?Windows-1252?Q?8HvxD+1oPRcd4FxrUu9MPSo9rndu3L4ifna/auKa/ZduepjLPgglkhED?= =?Windows-1252?Q?Ti6EcsTvuqN4ScmNjsCyYL9vq8IDGF2sRdr28CXIZ8UsJnroUFPk1XUx?= =?Windows-1252?Q?WNISez1ukIKIO2iFSFi9VdBgbiQB5gdxgUGAoO+RZsPjH1pQo2FETgXy?= =?Windows-1252?Q?mGLmtYgPPdNzXtrTi+zwWpzcPG8Z5G18K+cPJt46ZBiun7A6EQezGfMf?= =?Windows-1252?Q?MfR3kXzmt0utJGgSXtyS2bBhB0ptcnAQHpF5qCEQvMS860892RPumA+h?= =?Windows-1252?Q?Qzh6KqeygCFWrgcA1erWr0k0BMqZHFFVjWHntFOvMgccmFWaZxQEdgO2?= =?Windows-1252?Q?AShSXq51vXTmeVlS4lrSiIzztiugeRzXi0PALBZiY/m5rqlQrHN54oRt?= =?Windows-1252?Q?QoPdNbdhFcrVnzui+6s2mboX4Nj/1fvtr5+BIRk5oWxsekXIxTR8+ok6?= =?Windows-1252?Q?m0YRGXk6Pu2F6YceNS9SfhHKfawXebuQZ8aGICJoxGTnNeVQhKcq4+Tp?= =?Windows-1252?Q?OTJNq6MGynnyCI/cGBtE7yi1guyBSzlGeOpL+Lb+RZC98zxZ0nORkfaD?= =?Windows-1252?Q?RtOEvVXAm5i4MCPVBHQKyBazDFvXfcqi9nJ0RsP5SJG/nPfMB17auo70?= =?Windows-1252?Q?k1jjH4CYdAu9dWsYhojIYOXjvQP+S9CnQFyCbzUqMhjljSBU7WIVfJpG?= =?Windows-1252?Q?1BbtfwvsgK3QjZCuwmGrtLzGNVD9xhIg0Gui5X60+ZZCthE2isnT19XA?= =?Windows-1252?Q?SitFh5pXybrCVNpJuGKk0kzIrnDNpPh+2v25pP2PYMDRrMo6H7oDKdTL?= =?Windows-1252?Q?E8BRmn77Y5LuxZFG6c8l0V6FWZrZuS1p6pOmvv3h+X9AChF/DGNuEhFk?= =?Windows-1252?Q?cNcLGKuze+NkIw+yo+sK/ByloXxi6H8PtUweLWQyAzHEfWeXyGwkvjD0?= =?Windows-1252?Q?qZvWrfWN0grj8P8vYSMdvxcdXS+YDnnHtKvumxc7ZRydLL+sfIGp8qB8?= =?Windows-1252?Q?5vrs1S2X9ZJdhI5tehJ/0fVkmcZE+rZrI0bdEP1bnvWuSEuaaMhYgpQj?= =?Windows-1252?Q?cNpb1OUfYYoEvkkRm82cjZeznwo6u+wFukHi8wRH7DyxV8ITpmBtocVD?= =?Windows-1252?Q?Lk6jgHEuj8BnFFette1ASDvlC4qSuoVCTxbsEZUU5OGxpVaShdAb+bwy?= =?Windows-1252?Q?cTzD3hEMPho6OZPdVuIDc74/gFcme4v8SGi2gYzYJzQK2sUnU4eeCZDB?= =?Windows-1252?Q?2pAEEi+PVS5VEd7Vswy5V4cR1LLGl+xBacpagzf+FMKF1DHAlOcLoSgY?= =?Windows-1252?Q?S0EPOnxoR22j1FlaroGKWosZIHTVWKv4y86EEk52HBXps2tSQ/6krdWG?= =?Windows-1252?Q?Q13OPhiT8lxJ9NSzgNKix+gX+J5HpLSJabRAjzgNvyHm12JzxJiP4BTS?= =?Windows-1252?Q?kQAZ8x9QnQLDspbvM88IGE+JQCaofuHM7PLLWW4Xtc1QTPTXs8Q1jE0p?= =?Windows-1252?Q?a2u05Ch+3r0SUHRKiZrO1EirCsAiQbwlbEoHAYQZSY982FgLXJmDfYmD?= =?Windows-1252?Q?vEYfcUa0DzvWXZrH2S4bVPJro8y+ZU0ViRg+PQJdR0IlC8PxQwPCdZee?= =?Windows-1252?Q?i6uLS5T6Z6c1Lia/rXr86sb1XZQsaOvQgvBU3WjiI5jbJY5kfNgGR2tp?= =?Windows-1252?Q?JBmfmDytafdK+pd8ZqoL/ut+XIcy2b101Ic7p5EnFspOPLR1DuH59l+e?= =?Windows-1252?Q?KWoX1s5U8JkjF2mBFpaZbsOEs7+ABInX9miLlkmvYuPr0XyATHRzQmsN?= =?Windows-1252?Q?LI4x5jZY8jhhqKPiMMkdhGNmKLDmdm8hFGdYl1ZXHjq2JP49Tno963k2?= =?Windows-1252?Q?Q08ihYTQ2kQd4RcEcKqzQjCoWz/O+yynlBPKNTNmqKsXlE7/StQKjZ4A?= =?Windows-1252?Q?Bi7hzRR+4imFGXFU/0j0yxlPizszYCaKN65bFjweVClx86ygpyRRn9Er?= =?Windows-1252?Q?IWbAmW66gq7Dk7NzTc9Yrvm8MJ8Nm51HtiX4VAC5sBqoDXXzD73z6e6J?= =?Windows-1252?Q?NYdTXG+phKs=3D?=

freescout-helpdesk commented 2 years ago

Are you using modern Microsoft Exchange authentication to fetch emails?

afawaz2 commented 2 years ago

Yes, I am using oauth.

freescout-helpdesk commented 2 years ago

Oauth fetching is done by phpimap library. It looks like it had empty body issue in the past: https://github.com/Webklex/php-imap/issues/111

freescout-helpdesk commented 2 years ago

Will you be able to send an example of the email which is fetched with an empty body to the maintainer of https://github.com/Webklex/php-imap?

afawaz2 commented 2 years ago

yes I can

freescout-helpdesk commented 2 years ago

You can also send one such email to support@freescout.net. We'll try to check.

afawaz2 commented 2 years ago

done

freescout-helpdesk commented 2 years ago

We've just fetched your email using Webklex/php-imap library. And the body is present.

2022-05-17_10-46-11

Probably in your case the problem is in the server/PHP configuration. Are you using docker? What is your PHP version?

afawaz2 commented 2 years ago

I see you fetched my email, did you try to fetch the attached email? The attached .eml file is the one that failed. I am using the docker image from https://github.com/tiredofit/docker-freescout so docker.io/tiredofit/nginx-php-fpm:8.0

freescout-helpdesk commented 2 years ago

Try to upgrade PHP to 8.1 or higher.

afawaz2 commented 2 years ago

I switched one mailbox to using username/password instead of oauth and the problem stopped for that mailbox, the other mailboxes with oauth still have the empty email issue. I'll attempt the php upgrade to see if it resolves the problem.

freescout-helpdesk commented 2 years ago

Have you managed to solve the issue?

ollesson-allard commented 2 years ago

Hi i have de same problem with empty body from Microsoft Office 365 , @afawaz2 did you slow it with update the php to 8.1 ?

afawaz2 commented 2 years ago

I did not upgrade to 8.1 yet, we are still facing the problems with empty email bodies and missing attachments. I'll keep you posted when we do the upgrade.

ollesson-allard commented 2 years ago

Okey, With the latest Releases 1.8.17 some mail is working with but some is still empty body.

freescout-helpdesk commented 2 years ago

Any news on this?

freescout-helpdesk commented 2 years ago

Closing for no activity.

SecureCPU commented 2 years ago

Hey, I also have this issue where when I'm connected via the Microsoft exchange, I some blank email; as afawaz2 mentions, it mostly happens when emails are forwarded to me. I'm using a virtual linux server, with PHP 8.1.2

zhabi commented 2 years ago

hi i am also facing the same issue even using the latest version (1.8.22) by docker and m365 with oauth. i realized that the empty fetched emails have the different content-type from the original one on webmail, while those normal emails header is the same with the original one.

some example for reference:

e.g.1: most of those with empty content after fetched: content-type: text/html; charset=“gb2312” content-transfer-encoding: quoted-printable the original email: content-type: multipart/mixed or multipart/alternative

e.g.2: i tried to send email with rtf format, the original content-type is multipart/alternative, the fetched one changed to be text/html; charset=“us-ascii”. this time can be still get the email content

wishing this may be the hint to help.

ThaStealth commented 2 years ago

Experiencing the same issue, however I don’t have any new insights so @freescout-helpdesk let me know if you need me to try stuff, I'm running in Docker (1.8.31 on PHP 8.1), the problem started when I switch to Oauth for authentication (was on basic auth before)

salimnassim commented 2 years ago

We started experiencing the same issue recently, some of the fetched emails have an empty body even though the message can be read just fine with the PWA or alternative IMAP clients. Using Office 365 with OAuth.

jeliasson commented 2 years ago

We have the same problem, and the only thing we notice is this error in the Fetch Error logs. Not sure if it's related.

image

Freescout version: 1.8.31 PHP version: 8.1.11

freescout-helpdesk commented 2 years ago

We have the same problem, and the only thing we notice is this error in the Fetch Error logs. Not sure if it's related.

image

It may help to track the problem. Can you send full error text from Manage > Logs > App Logs

jeliasson commented 2 years ago

It may help to track the problem. Can you send full error text from Manage > Logs > App Logs

Unfortunately there is nothing that seems to be of interest in app Logs. Granted, during testing and troubleshooting we cleared logs in Fetch Errors so we might just have deleted it. But reading the error message more carefully it tells us that 1st parameter (which I would assume would be the body in this case) is null, so I guess the overall problem is still that the e-mail body sometimes being null/empty.

freescout-helpdesk commented 2 years ago

If someone having this issue can provide SSH access to your FreeScout server, please email at support@freescout.net. Also send an example of the empty conversation.

Reubenwelsh commented 2 years ago

We are also having this same issue, some emails work fine, others don't load at all.

Error started when we moved to SMTP AUTH via O365 - roughly 50% of emails work 50% end up empty. Tried different clients, different email providers sending but cant find a pattern.

mbl72 commented 2 years ago

Same issue here. I am able to provoke some of the issues from my email accounts.

An email without any special charactors (like the danish æøå) will come through, while another with the characters will end up with a blank conversation.

From another account, I'm able to provoke the crash in FetchEmails.php line 988. Again related to the danish characters.

Funny thing, however. If a collegue of mine does the same thing - I passes without issues.

I've sent the raw eml files from Outlook to support@freescout.net

freescout-helpdesk commented 2 years ago

Same issue here.

Corresponding issue in Webklex/php-imap library: https://github.com/Webklex/php-imap/issues/230. Try to pind the developer of the library or provide him with examples.

Funny thing, however. If a collegue of mine does the same thing - I passes without issues.

What email client does your colleague use?

mbl72 commented 2 years ago

We both use Outlook through Microsoft 365 - With ongoing updates - Current Channel.

I just did a test from the same account that caused the crash - but now from my phone (android) - Gmail client. And that one went through without any problems.

freescout-helpdesk commented 2 years ago

We've updated webklex/php-imap lirary to the latest version. Try to update to FreeScout v1.8.35 and see how it goes.

freescout-helpdesk commented 2 years ago

We'll close this for now. Let us know if this issue still persists in FreeScout v1.8.35

BikeyBoi commented 2 years ago

After updating to v1.8.35 the problem still exists and we still recive empty messege body.

mbl72 commented 2 years ago

Just a thought.. I would be nice, if the failure log could contain the title from the failed mail - That way it would be easier to see which mail actually failed.

Furthermore it would be great, if a filter of some kind could be added to the "Logs Monitoring" under Fetch errors, right now we have a lot of "Connection Errors", every day.

this prevents us from using the alert routine..

From the log: Nov 25, 06:09:02 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)
Nov 25, 05:17:03 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)
Nov 25, 03:30:04 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)
Nov 25, 02:44:06 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)
Nov 25, 02:19:35 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)
Nov 25, 01:51:25 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)
Nov 25, 01:50:37 Error fetching email Error: connection setup failed; File: /www/html/vendor/webklex/php-imap/src/Client.php (391)

mbl72 commented 2 years ago

I just updated to 1.8.36 and did a "Fetch EMails" via the system - Tools menu for the the last 14 days. This created a lot of new conversations. The version seems to have fixed the "Passing $null to the..."-problem. All conversations, however, doesn't have any text in them.

zhabi commented 1 year ago

Updated to 1.8.36 by docker but the issue is still existing. Those email still with empty content

nrukavkov commented 1 year ago

The same problem. Could anyone fix it?

freescout-helpdesk commented 1 year ago

Here is the corresponding issue in the webklex/php-imap library: https://github.com/Webklex/php-imap/issues/275

ghost commented 1 year ago

This is an issue we've been experiencing for some time too. Our emails are fetched from an Exchange Online server, but I'd say about 30-40% of them are blank. Looking at the source, they usually show:

<br>
<br>

or

<div><br></div>

Attachments are fetched correctly, but having the manually edit the source and paste in the original is quite tedious.

Are there any logs or information I can provide to help with this being resolved?

freescout-helpdesk commented 1 year ago

@joseph-arun can you send a source code of one of such emails from your mail server?

ghost commented 1 year ago

Sure, I have sent source code to your support email address. Thanks

freescout-helpdesk commented 1 year ago

Developer of the webklex/php-imap library asked to send him examples of emails which have been fetched with empty body: https://github.com/Webklex/php-imap/issues/275#issuecomment-1379906628

jeliasson commented 1 year ago

A fix seems to have been merged in upstream Webklex/php-imap, or are these unrelated?

freescout-helpdesk commented 1 year ago

A fix seems to have been merged in upstream Webklex/php-imap, or are these unrelated?

Yes, but people still continue to receive empty emails when using OAuth IMAP authentication.

jeliasson commented 1 year ago

@freescout-helpdesk My bad. I did not pay attention that the merge was in October.

SecureCPU commented 1 year ago

Hey, Don't know if this helps. I used to have this issue a lot. I changed up some settings and haven't had this issue in a long while.

(I've also disabled Microsoft Azure Security Defaults: https://help.protectedtrust.com/enabling-security-defaults-for-azure-active-directory-in-office-365)

Here are the settings I use:

I first disconnected the connection in FreeScout. Then deleted the Azure app and settings and created the connection from scratch using the following settings.

Screenshot_20230117_0952231 Screenshot_20230117_0953541

Hope this helps!

salimnassim commented 1 year ago

Hey, Don't know if this helps. I used to have this issue a lot. I changed up some settings and haven't had this issue in a long while.

(I've also disabled Microsoft Azure Security Defaults: https://help.protectedtrust.com/enabling-security-defaults-for-azure-active-directory-in-office-365)

Here are the settings I use:

I first disconnected the connection in FreeScout. Then deleted the Azure app and settings and created the connection from scratch using the following settings.

Hope this helps!

Using broader API permissions fixed the empty message body issue at least for now. Thanks for the solution!