Performance - sendouts taking many hours

[bossanova808]

Question

As you know, I love just about everything about Campaign :)

The one thing I am not super in love with is the speed of sending. On average, a mail-out to our list (~9000 contacts) takes distinctly north of 6 long hours. (Obviously not comparable, but with Mailchimp it is like 15 seconds ;) )

We're running on a $20 Vultr VPS that is more than adequate - for our general serving needs at least. It's juts a bog standard Ubuntu LAMP setup, managed by ServerPilot.

For background, I use a PHP CLI / queue runner approach ( using systemd per https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms#solution-4-utilizing-systemd)

Crontab:

# Cronjob for Campaign plugin - moving off the hour here as queue errors... 
# https://github.com/putyourlightson/craft-campaign/issues/143
5,17,25,35,49,55 * * * * cronic php /srv/users/serverpilot/apps/whatever/craft/craft campaign/sendouts/run

I have 1024mb allocated in the Campaign settings. Max batch size currently 1000.

I have also set a very large TTR in my app.php component bit:

        'queue' => [
            'class' => craft\queue\Queue::class,
            'ttr' => 60 * 60, // Minutes * seconds, i.e. 60 * 60 = one hour
        ],

However, the TTR doesn't see to take - with my latest sendout, in the queue manager in the Craft CP it still reported 300 for the Campaign send-out jobs.

For PHP, I had set a timeout of 0 as I thought my CLI use/TTR would be ok with this. However, I got:

2021-03-18 18:29:47 [-][-][-][error][craft\queue\QueueLogBehavior::afterError]  [244375] Sending “2021-03-18 - Massive BenQ Easter Monitor Sale!” sendout [batch 2] (attempt: 1) - Error (time: 1616052587.082s): The process "'/opt/sp/php7.2/bin/php' '/srv/users/serverpilot/apps/imagescience/craft/craft' 'queue/exec' '244375' '300' '1' '0' '--verbose=1' '--color='" exceeded the timeout of 300 seconds.
2021-03-18 18:29:47 [-][-][-][error][Symfony\Component\Process\Exception\ProcessTimedOutException] Symfony\Component\Process\Exception\ProcessTimedOutException: The process "'/opt/sp/php7.2/bin/php' '/srv/users/serverpilot/apps/imagescience/craft/craft' 'queue/exec' '244375' '300' '1' '0' '--verbose=1' '--color='" exceeded the timeout of 300 seconds. in /srv/users/serverpilot/apps/imagescience/releases/dev/vendor/symfony/process/Process.php:1213
Stack trace:

...in my console.log - showing that YII is still using the 300 as the TTR (as per the CP).

This meant my latest send-out stalled.

So I switched back to 180 Time Limit in Campaign and (nervously) restarted the send-out (praying it wouldn't start from scratch and re-email the initial batch - and indeed it didn't - I should never have doubted you ;) ) - all resumed well and the slow, steady process was fine from there.

The email service (having just switched) is Postmark. The results were the same with Mailgun, no appreciable difference in speed was observed really.

My questions are:

Have I missed something here? Is this sort of time-scale common or is it running much slower than normal for me because I have or have not done something I should have? From a business perspective it would be a lot nicer if it was more like 1 hour than 8ish....but I've got no real way of gauging what is normal/realistic.

I'd expect the TTR in the queue manager to show 3600, not 300. Is there some way I can confirm that other than looking at actual queue jobs?

One thing I do need to do is add more caching to the newsletter templates. So there's definite room for improvement there...will that dramatically speed things up though? These are pretty simple templates...although using craft-mjml so perhaps there is substantial hidden time in that.

[bencroker]

For reference, the recommendations given in this section of the docs are relevant: https://putyourlightson.com/plugins/campaign#sending-to-large-lists

The main contributor to a slow sending process is the rendering of the email template, as this has to be done for each unique contact. So the more of the template you can cache using {% cache globally %}, the better. Also, be sure to avoid using the MJML API to render email templates as per the docs:

Using the MJML API to render email templates will significantly slow down the sending process due to latency, therefore we highly recommend using MJML only to generate the markup for your templates.

You could also increase your batch size to 10,000 to avoid delays between sending batches, which may also help. With the right measures I expect your sendout to complete within an hour.

[bossanova808]

Thanks, Ben & yep, have read that, hence the efforts above.

Any thoughts on the TTR side of things?

But yep, I will cache absolutely everything that is non-dynamic! I figured most of the delay was coming from the Campaign <> API interaction as the templates are really pretty simple but sounds like that is very much the key thing to look at then.

I am using the MJML plugin with {% filter mjmlCli %} - that uses the MJML CLI rather than the API. This is what you mean, right - i.e. the API is the thing to avoid. Or are you proposing grabbing the output from MJML and putting all that in the templates?

[bencroker]

The TTR is much less relevant than you may think. It is the time to retry when a queue job doesn't respond, so 300 is a safe value to have it at.

Avoid using MJML Twig tags in your templates, by grabbing the raw output and putting it in the email template, for the most optimal results.

[bossanova808]

hmm, ok will have to look in to that. But given we're using a matrix block builder approach, that's going to make for some messy templates....

[bencroker]

Watch the MJML video at https://putyourlightson.com/plugins/campaign#mjml to see a clean way to do it.