Email replies not being tracked (possible fix using imap_fetch_overview instead of imap_search)

[tslater]

What type of report is this: Bug Report

Description:

Emails are not tracking in mautic.

PHP Version: PHP 7.0.27 (cli) (built: Feb 17 2018 01:53:37) ( NTS ) Copyright (c) 1997-2017 The PHP Group Zend Engine v3.0.0, Copyright (c) 1998-2017 Zend Technologies

Mautic version: 2.12.2

Steps to reproduce:

1. Setup IMAP for a gsuite account
2. Make sure you have replies in your tracked folder
3. Run the command: php /var/www/html/app/console mautic:email:fetch

Result: 0 emails processed for {imap.gmail.com:993/imap/ssl}INBOX with a criteria of UID 20:* Expected: 2 emails processed

I've done a lot of digging and I believe that it comes down to one simple issue: usage of php's imap_search() function in the searchMailbox function of the Mailbox.php class as seen here: https://github.com/mautic/mautic/blob/staging/app/bundles/EmailBundle/MonitoredEmail/Mailbox.php#L559.

If you notice, the criteria outputted by the logging for the fetch command is UID 20:*. From what I can tell, this isn't supported by php's imap_search() function: http://php.net/manual/en/function.imap-search.php. It isn't in the list of criteria, and I tested it extensively, with many variations in the php interactive console. I was able to get messages via the "all" command, and several others, but not with the undocumented "UID" command.

I did have success with imap_fetch_overview() http://php.net/manual/en/function.imap-fetch-overview.php, however. I propose that we use imap_search() for searches, and imap_fetch_overview() for uid range-based lookups.

[xtjoeywx]

I have been trying to figure this out for a while now. Please make this one a priority fix. This would save me a TON of time during the day of manually removing people from campaigns because they replied. I feel this also is the reason why mine is not recording bounces.

[tslater]

@dbhurley I got pretty deep into the code, and realized that although I understand the problem, I'm not familiar enough with the code to where I feel confident I could contribute a proper solution. I noticed that you thumbs-ed up the issue--and that you're a contributor. How familiar are you with this area of code?

[luizeof]

Hi guys ... We heavily test this feature with: GSuite, Office365 and Zoho.

Mautic read the email reply but do not register the reply on database on all cenarios.

root@f20b5bbf3ec4:/var/www/html# php /var/www/html/app/console mautic:email:fetch 1 email processado em {imap.zoho.com:993/imap/ssl}INBOX com o critério UNSEEN 0 e-mails processados em {imap.zoho.com:993/imap/ssl}INBOX com o critério UID 15026:*

[tslater]

@luizeof, I feel confident I understand the issue, it's pretty plain from the code. I just don't have the confidence/time to get comfortable with the code to make a fix yet. I've never dev'ed on mautic before.

[miteshgala]

Glad I finally found this thread. I have spent a ton of time trying to understand how this works and why it doesn't. I will try to look at the code next week but am also not familiar with the code base and have to see how steep the learning curve is.

[afonseca08]

I agree this is a high priority fix and will be following this issue.

[thedude83]

Any progress on this? I need this functionality for my campaign.

[xtjoeywx]

I second the above comment. I also really need this for Mautic to work the way I need it to. It takes me hours to manually remove people from a campaign when they reply. I know I can have them removed by them simply opening the email, but that doesn’t get us customers. In our business we see more results from the 4th or 5th email (the one they reply to). I want them to receive emails until they either reply or unsubscribe. We use GSuite because of all of the tools it has and I’m sure a ton of people do as well so I’m surprised this isn’t a larger issue.

[Dean-Cookson]

Just started using Mautic specifically for this function as no other platform offers it and as everyone else says, it takes forever to do this manually. Would be great if someone could fix this problem.

[dimitrivisser]

Is this problem not fixed yet? Cannot get it to work here.

It would be better to delete this function from Mautic if it doesn't work. It took me hours of testing, and now I read that it simply will not work.

[luizeof]

@dimitrivisser did you develop with php ?

[dimitrivisser]

@luizeof Yes, for many years. Why ?

[afonseca08]

I'm curious, anyone know if is this feature is broken on the hosted version offered by Mautic.com as well? If not, do they contribute fixes back to the open source version?

[luizeof]

@dimitrivisser I believe this problem is not in their roadmap, could you please help with the correction?

[Dean-Cookson]

I was looking at buying their hosted version a couple of weeks ago and raised this. They suggested that they were working on it and it was maybe a month away from a fix - this may have been sales tactics but they also said I wouldn't need to pay until the solution was ready.

[alanhartless]

Yes, we have this as priority to look into. It seems that UID n:* is not working as expected with imap_search.

[luizeof]

@alanhartless great news! When you commit the fixes I'll test on same day ;)

Thanks for your great work!

[xtjoeywx]

@alanhartless @dbhurley I was really hoping this would be fixed on this new 2.14 version. Is there any chance it could be soon?

[LevryKurniawan]

I found the same issue in version 2.12.2 and resolved in 2.13.1. But I found a new issue in 2.14.0. #6369

[filchakov]

I have the same problem in v2.14.1-dev.

I try to read email for tracking reply via IMAP.

My PHP version 7.1.19

php /var/www/html/app/console mautic:emails:fetch -env=prod 4 emails processed for {imap.yandex.com:993/imap/ssl}INBOX with a criteria of UNSEEN 0 emails processed for {imap.yandex.com:993/imap/ssl}INBOX with a criteria of UID 4:*

[alanhartless]

PR to fix this coming today. Sorry for the delay guys.

[alanhartless]

The fix is in https://github.com/mautic/mautic/pull/6376

[sudo-shubham]

@alanhartless Any updates when this issue will be merged? There is still one review pending.

[npracht]

Hi @sudo-shubham we're considering as candidate for 2.14.2 version :)

[janwinkler]

Issue seems to still exist. Im using 2.16.0 and get output: app/console mautic:emails:fetch -env=prod 1 email processed for {mymailbox.com:993/imap/ssl}INBOX with a criteria of UNSEEN but in contact details there is no change or anything that tells us that the contact replied.

[Anonyma]

^ I'm still having this issue on 2 different Mautic installations and 2 different monitored e-mail addresses (both from IONOS). One is Mautic version 2.16.2 and the other 3.0.1.

The campaign doesn't seem to detect the contact replies, and no new log is registered in the contact view's timeline indicating a contact reply.

(I get the "success!" message in the settings tab, and when I manually execute mautic:email:fetch I also do get the expected "X e-mails processed" message (so no errors). I also tried manually setting the credentials in the "Contact Replies" section, and replying from addresses hosted on different platforms but it still doesn't work)

EDIT: Posted in the support forum

[successful-fella]

Can someone confirm if this feature broke or working? For the first time, it tried fetching UNREAD criteria and after showed UIDs with success but no data gets inserted in table email_stat_replies.

[RehanNischal]

Is it solved yet?

[FKrafft]

I have a similar issue, Wanna track down what is off. Considering the UID is off. Does anyone know where I can print out UID, and see what search paramers are being using for imap serach. A hint of what I should search manually in the db?

[reegen66]

So we have 2023 and we still have the same problem. Email processed as UNSEEN then nothing happends, this event is not registered in mautic, so therfore you can't trigger any action based on this event. This very important issue. I do really appreciate such a great piece of code for free, but if this is not working i won't be purchasing your solution in the cloud.

Best, Piotr

[npracht]

Hi @reegen66, On my side my customers are using it and it works properly.

You should check your configuration. Be sure to set these sections, if empty it doesn't work.

[reegen66]

Thank you @npracht , i was not aware that bounces needs to be set in order to make replies work.

I will check it out!

Have a good day!

[IdeaSystems]

Hello friends!

I believe I'm experiencing the same issue with my Mautic installation. The email box is readable but Mautic does not understand that the email was either opened, read, or rejected. Has anyone found a solution to this problem?

Is Mautic supposed to understand that the email has been replied to as soon as it reads the correct inbox? I must do something else on my side. Cron tasks have already been configured.