Sending Speed & "mailcoach-feedback Job" size

[jab1000]

Hi Spatie,

For the past 5'ish emails, we have created a "campaign" and sent to X list at around 100k in size.... then when that one is done (zero send jobs) we will queue up the email to another list of 100k'ish. Overall the list of 100k will send in about 5-6 hours overall.

This time I decided to send 1 and once all "sends" were done spooling to the "send-mail" job, we fired up the 2nd one. After several hours it still had 170k jobs -- when I just checked it now (about 10 hours later) it has 86K emails still to go. Plus the "mailcoach-feedback Job" has been sitting at about 7,000 with "about an hour" to process..... As I started doing more investigation our database has been pegged at 95% CPU since not too long after.

After some digging via Google I noticed a closed ticket on "Large mailcoach-Feedback job" and they deleted LOTS (200k'ish) of records in "webhook_calls" -- I did the same and now in the past 20ish minutes the "mailcoach-Feedback" job has gone down to zero.

Weird thing is the database percentage is still around 90%.... and with Covid19 it isn't customers unfortunately.

Server details: MySQL - AWS RDS - R4 Large instance Email - SES with a 1M/24 hour quota at 90/sec.
Large T3 instance running mailcoach as standalone - averaging around 25%

Does seem like something is potentially funky on "webhook_calls" since when deleting it does speed things up quite a bit.

My mailcoach "throttle" is set to 75 --> "theoretically" then 4500/minute and "theoretically" about 50 minutes to send 220k emails -- challenge is now at 80k emails to send with an ETA of 4 hours.

Thoughts or things I can test to speed it up?

[AlexVanderbist]

Hi @jab1000,

I'd hoped to respond sooner but we were looking to tag a fix for this particular issue. Unfortunately we're waiting for some other issues before tagging this fix in v3 of the package when Laravel 8 is released this summer.

As you've mentioned the webhook_calls table can slow things down quite a bit. The quick fix for this would be to delete processed webhooks after they've been processed for some time. However, certain webhooks might arrive more than once. Here's two possible solutions for cleaning up webhooks you can implement yourself:

⚠ You'll need laravel-mailcoach-ses-feedback version 2.2.0 or higher

1. Deleting webhooks right after they've been processed (easiest fix but might register some mail events multiple times)

• register the following listener for the Spatie\Mailcoach\Events\WebhookCallProcessedEvent in your EventServiceProvider's boot method:

  public function boot()
  {
  Event::listen(\Spatie\Mailcoach\Events\WebhookCallProcessedEvent::class, function ($event) {
      $event->webhookCall->delete();
  });
  }

2. Deleting webhooks that have been processed for > 30 minutes to avoid processing a webhook more than once

• create a migration to add a processed_at timestamp column to the webhook_calls table
• register the following listener for the Spatie\Mailcoach\Events\WebhookCallProcessedEvent in your EventServiceProvider's boot method:

  public function boot()
  {
  Event::listen(\Spatie\Mailcoach\Events\WebhookCallProcessedEvent::class, function ($event) {
      $event->webhookCall->update(['processed_at' => now()]);
  });
  }

• add a clean up command that queries old webhook calls and deletes them: https://github.com/spatie/laravel-mailcoach/blob/5befb9a517de8e716a460162f3424f221068a6c0/src/Commands/CleanupProcessedWebhookCallsCommand.php (you might want to decrease the 1 hour default to 30 minutes or something)
• finally, schedule this command to run every 5 minutes or so

This should speed things up considerably until we implement the fix ourselves in v3, later this year. Either way, this bottleneck is something we've got on our radar and will keep looking into as development moves forward :)

Good luck!

[freekmurze]

Thanks @AlexVanderbist

[electronick86]

Hello All.

I was doing some test concerning the send performance. I create a list of 5000 recipients. Then I opend a fake smtp server (michielbdejong/mailtrap, This Docker image will listen on port 25, and do nothing else than responding to SMTP traffic in a valid way).

I set my send rate to 100/sec into config/mailcoach but my effective send rate was about 30 mails/sec even if there was no feedback to process.

I also increased the number of process into config/horizon to 100 (or even 400) but when the campaign was being sent, the horizon dashboard shows that there was 20 processes for the feedback; 20 processes for mailcoach, etc. There was only 50 processes for the send mail.

After that, I replaces my config/horizon.php like that and I saw a huge difference. Like we see on the last line of my screenshot, I sent 102 emails a second (=the value of the throttle)

//            'mailcoach' => [
//                'connection' => 'mailcoach-redis',
//                'queue' => ['send-campaign', 'send-mail', 'mailcoach-feedback', 'mailcoach'],
//                'balance' => 'auto',
//                'processes' => 400,
//                'tries' => 2,
//                'timeout' => 60 * 60,
//            ],
            'mailcoach2' => [
                'connection' => 'mailcoach-redis',
                'queue' => ['send-campaign',  'mailcoach-feedback', 'mailcoach'],
                'balance' => 'auto',
                'processes' => 10,
                'tries' => 2,
                'timeout' => 60 * 60,
            ],
            'mailcoach3' => [
                'connection' => 'mailcoach-redis',
                'queue' => ['send-mail'],
                'balance' => 'auto',
                'processes' => 400,
                'tries' => 2,
                'timeout' => 60 * 60,
            ],

In the end, I probably don't need 400 processes, but just to share the fact that it's maybe a good idea to specify a number of processes just for one queue like so.