search

TheDMSGroup / mautic-eb

Mautic + Elastic Beanstalk = Scalable marketing automation
https://www.mautic.org
GNU General Public License v3.0
105 stars 27 forks source link

Have you experienced EFS timeouts during bulk emailing? #112

Open oytuntez opened 4 years ago

oytuntez commented 4 years ago

Hi team and @heathdutton,

This is not about your repository, but we have quite a similar setup for Mautic. We are using file spool and we are experiencing not only a horrible performance, but also intermittent timeouts especially with a larger queue (20K, 30K messages in spool directory). EFS already takes at least 10 seconds to list the directory, which I believe is causing 451 timeout in the SMTP protocol.

I hate both file and memory spools, how are you managing this? I am going to experiment with some Redis hack to force Swiftmailer to use Redis as spool, but first, I have 30K emails stuck in the queue at the moment...

Any insights are welcome.

heathdutton commented 4 years ago

EFS for email spool is really just a stop-gap to get you rolling with an initial setup. I wouldn't recommend that at all for long-term use. We typically have very little in the spool because we're offloading as much as possible. We've got SES for mautic direct (forgotten password, user stuff), and Mailgun, Sendgrid and others for the campaign messaging using the "Contact Client" plugin. Anyway... I recommend using API based mail services as much as possible. With your volume, you need to avoid SMTP like the plague.

oytuntez commented 4 years ago

You have some amazing plugins under DMS which I am diving into soon!

We are using Mautic for transactional emails as well, and I want to keep using SES. How would you integrate this? Instead of using queue for emails, should I simply pick immediate?

I've been waiting for this (currently no bandwidth to undertake the PR myself): https://github.com/mautic/mautic/pull/6977

Yesterday night, I was hard debugging and found 2 performance issues with both Mautic and Swiftmailer. They both used DirectoryIterator which caused bottleneck in NFS environments. Using opendir + readdir solves the issue to a big extend.

Now finalizing these findings and fixes on my local and staging envs.

oytuntez commented 4 years ago

Here is the issue I opened on Swiftmailer: https://github.com/swiftmailer/swiftmailer/issues/1241