problem with SMTP?

[viriatusX]

Hello! I have used Ghosler in real production for the first time. Many users do not receive the email. I attach the "error logs". I used for this: Gmail with port 465 (TLS).

[2023-12-24 10:28:36 UTC] => [ERROR] => Newsletter: Error: Data command failed: 421-4.3.0 Temporary System Problem. Try again later (10). For more information, 421-4.3.0 go to 421 4.3.0  https://support.google.com/a/answer/3221692 g3-20020a05600c310300b0040c6b2c8fa9sm21226207wmo.41 - gsmtp
at SMTPConnection._formatError (/home/juani/Escritorio/node_modules/nodemailer/lib/smtp-connection/index.js:790:19)
at SMTPConnection._actionDATA (/home/juani/Escritorio/node_modules/nodemailer/lib/smtp-connection/index.js:1678:34)
at SMTPConnection.<anonymous> (/home/juani/Escritorio/node_modules/nodemailer/lib/smtp-connection/index.js:1650:26)
at SMTPConnection._processResponse (/home/juani/Escritorio/node_modules/nodemailer/lib/smtp-connection/index.js:969:20)
at SMTPConnection._onData (/home/juani/Escritorio/node_modules/nodemailer/lib/smtp-connection/index.js:755:14)
at SMTPConnection._onSocketData (/home/juani/Escritorio/node_modules/nodemailer/lib/smtp-connection/index.js:193:44)
at TLSSocket.emit (node:events:513:28)
at addChunk (node:internal/streams/readable:324:12)
at readableAddChunk (node:internal/streams/readable:297:9)
at Readable.push (node:internal/streams/readable:234:10)

[ItzNotABug]

That's a very good number of users for trying Ghosler 😉.

---

If you observe the logs, you'll see this -

Data command failed: 421-4.3.0 Temporary System Problem. Try again later (10).
For more information, 421-4.3.0 go to 421 4.3.0  https://support.google.com/a/answer/3221692

A quick search revealed this page - https://support.google.com/a/answer/3726730 which shows the error codes, also the one shown in the logs. 421, | "4.3.0", Temporary System Problem. Try again later.

This doesn't seem like a problem with Ghosler but with Gmail & SMTP.

[ItzNotABug]

I also notice that this can also be caused due to sending bulk emails via Gmail. See - https://stackoverflow.com/a/55358989/6819340 & https://stackoverflow.com/a/39108563/6819340.

Seems like I'd have to add a sleep timer logic. But given the amount of users, it might take that much of seconds to send emails.

[viriatusX]

I also notice that this can also be caused due to sending bulk emails via Gmail. See - https://stackoverflow.com/a/55358989/6819340.

Seems like I'd have to add a sleep timer logic. But given the amount of users, it might take that much of seconds to send emails.

It doesn't matter if it takes longer, as long as everything is sent correctly 🙏😁.

[ItzNotABug]

I'd agree on that to some extent but make sure to check that Gmail SMTP also has strict limits. See: https://support.google.com/a/answer/166852.

[viriatusX]

Absolutely right... We would have to look for a SMTP server with larger limits.... Without this, the tool (in my case) would not be very valid :(

[viriatusX]

I will try the MailRelay SMTP server: https://mailrelay.com/es/precios/ Limit of 80K emails per SMTP / month. More than enough.

[ItzNotABug]

I think if we use a sleep based logic, it will take a lot of time to send all emails.

Assuming the same number of subscribers as yours i.e 1,292, and sending op. might take around 1 to 2 seconds, add the delay on it of roughly 1 second. The sending time goes over 30 minutes.

I don't think this is feasible, performance wise, because Gmail has a strict sending limits.

[viriatusX]

Yes, I totally agree with you. I close the issue as it is not a Ghosler error.

To upgrade to the new version is done with "ghosler install"?

[ItzNotABug]

@viriatusX ghosler update to update the app to the latest version.

[viriatusX]

Hi @ItzNotABug

I am not opening an incident as it is more of a query than an incident. I have contracted AWS SES as my SMTP provider. I have a limit of 50K/day and 14 emails per second. Something curious happens. Ghosler has managed to send 588 of the 1352 emails correctly.

I ask with ignorance. I have seen that Ghosler uses nodemailer. Is this configured in your program as infinite? https://medium.com/@susanne.lundkvist/using-nodemailer-to-send-bulk-emails-cbad289fce0e

I don't know if Nodemailer will be blocking so much mass mailing as well.

Ghosler shows this error:

[2024-01-08 17:59:39 UTC] => [ERROR] => Newsletter: Error: Message failed: 454 Throttling failure: Maximum sending rate exceeded.

It shows as if it is checking the AWS SES limit, but this is not the case. Googling I see this article from 2015 (https://github.com/nodemailer/nodemailer/issues/436) also about the same "pool" issue.

If nodemailer is not the problem. Is there a way for Ghosler to detect the members that did not receive the email and do a "retry". If it detects that there are 1352 members (in my case) but only 588 emails sent. I don't know if it detects which emails could not be sent, and proceed with a retry. Or adding some "queue" for emails.

Thanks!!

[ItzNotABug]

Hey @viriatusX Thanks for flagging this.

I think it is actually a throttle limit by SES & I also guess that similar limits might be added by other major players too. Looking up on similar usecases, I see that a way to manage this could be by using rateLimit flag to the underlying transporter.

I'll investigate more on this.

[ItzNotABug]

I don't think we need the pool flag as it it involves setting up maxMessages per connection & the rateLimit flag is also marked @deprecated. My current choice is to introduce a batching logic with a defined delay.

---

Flow:

1. Assuming 1500 subscribers
2. Splitting the subscribers further into batches, default 10 = 150 Batches
3. After sending email on each batch, we wait via a delay, default 1000ms (1 second)

Note: This introduces a delay in sending the emails but this is manageable and isn't too exponential unlike our previous discussions.

---

Both these options (emails per batch, delay between batches) can be configured via settings, for each mail configuration.

[ItzNotABug]

Hey @viriatusX, Please see and test the email-batching branch if you can.

I've tried testing on a small number of users and it seems to work fine, however for your use-case with SES & a big subscriber number, we can have a good idea if this works fine.

Feel free to let me know if thats not possible, since this would still require you publishing/re-publishing a post on your site.

I'll also be closing this issue & creating a new one as this one is pretty backdated & corresponds to a previous issue.