Performance and bulk sending

[raulp]

Hey @freekmurze

I finally started to play with this more over the weekend. So far all i can say is that i love it.

I watched all your videos and read all docs. Everything looks clear. Good job on that!

I have one hard question regarding the sendout process: why was the decision made to create a job for each recipient when most of these providers out there allow you to send 1 API call with 1 body and hundreds of recipients?

Let me explain why i think this is important. For a list of 50,000 people that get mailed daily, it's ok if the mail takes 1hr to process and send. For a list of 2mil people where i need to mail them daily 3 times, this might not be good.

Another point here: for every request, you have to compose and send the body (and run 2-3 SQL queries) and queue the job. That takes data. And when sending the API request, that takes network data. Again, for 50,000 people once a day, not a problem.

Now, i can think of 2 reasons why this decision was made:

1. you heavily relied on Laravel's Mail, which is single-user based
2. maybe not all providers return messageId after each send and all the feedback packages are relying on that to track the subscriber

I understand this might not be the scope of the package. And that is fine, not to many people want to use this to the extreme. I wonder if you can point me in the right direction so i can try to build that as a module, that can be enabled later if you want on a per campaign basis maybe. Imagine later there might be a feature where you can send campaigns across all your lists (which i think should be a feature request). And you can choose if you want to use regular mailer (slower) or bulk mailer (faster).

Yes, i know some of these ESP's have limits, but the limits are very very high (like . The only one that is limited right now is Amazon SES, but even that one, after a short period of time, you can get to 100 emails per second. With the current system, to send 100 emails per second we probably need 30-60 workers on the queue.

I've coded a few drivers for all these providers for work, where we send over 2mil emails per day for our customers using their own different gateways. And i want to build them for myself anyhow, even if they will be added in your package or not (which is totally ok). I just want to know, if you would to code this, how would you do it (from a coding overview using your package)

Thanks man, again, great work on the package. I love the flexibility and can't wait to dig in and expand it.

[ryancwalsh]

most of these providers out there allow you to send 1 API call with 1 body and hundreds of recipients?

Good point!

[freekmurze]

I finally started to play with this more over the weekend. So far all i can say is that i love it.

Thanks, glad you like it!

I have one hard question regarding the sendout process: why was the decision made to create a job for each recipient when most of these providers out there allow you to send 1 API call with 1 body and hundreds of recipients?

We wanted to make the code very simple and easy to restart should there be any problems while sending. If you have an API and you sent your mail to multiple recipients, and that call fails, it could be hard to determine which mails have been sent, and which not.

Not only do we have a job per mail sent, there's also a row in the db campaign_sends for each mail that has to be sent. We can use this to track mails that have failed and also make certain no mail is sent twice.

This all would be much harder to do if a single API call would send multiple mails.

In general, I prefer code clarity over performance.

I understand this might not be the scope of the package.

Such high volume cases are not our focus. We designed Mailcoach for email lists that have a couple of thousand, maybe tens of thousands subscribers. For those use cases Mailcoach will work fine. Maybe it'll work fine for millions of subscribers too, I honestly don't now. If there are specific problems, I believe we have enough escape hatches (like the ability to override actions) in place, so use can modify the default behaviour.

I've coded a few drivers for all these providers for work, where we send over 2mil emails per day for our customers using their own different gateways.

If you seen performance improvements that could be applied without sacrificing on code clarity or restartability, feel free to propose specific changes.

[ryancwalsh]

@freekmurze I appreciate all the thought you have put into this!

[electronick86]

Hi All,

Several months later, I reopen this discussion because, maybe, you have some more feedback from users.

I create several project for clients based on Mailcoach and I really loved it. I learned (and learn) a lot when using your package. Thank you very very much!

The biggest mailing list at the moment contains 200k emails and I works like a charm. I wonder if it's a bad idea to plan using Mailcoach with subscribers lists even bigger. I mean 500k - 800k.

Do you have good customer feedback (or bad experiences) with this kind of sizing? What's the best place to share that kind of experiences? Is a topic like this one a good place?

Thanks again Spatie 👍

[freekmurze]

I think the biggest one that I know of is 400K subscribers and there were no problems whatsoever. Feel free to converse here with anyone that responds to this issue.

[electronick86]

Hello @freekmurze,

Thanks. I'll let you know how does it works in the futur. If I reach 500k, do I get a trophy? ;-)

[freekmurze]

Here you already go! 🏆