Add multiple Email (SMTP) services for more stable Email sendings

[Clemens85]

Scenario: Sendgrid deactivated currently the account, hence no further mail sending is currently possible till it is re-activated.

Goal: Reduce single point of failure with SendGrid and integrate more email service providers like AWS SES and MailJet (and maybe also traditional mail boxes like Gmail).

Tasks:

• [x] Create account on MailJet

• [x] Checkout AWS SES infrastructure setup

• [x] Enhance infrastructure setup for properly setting up DKIM, SPF and DMARC entries

• [x] Think about possible mail sub-domain sender names, like e.g. post.runyourdinner.eu or message.runyourdinner.eu or mailing.runyourdinner.eu, contact.runyourdinner.eu, Info.runyourdinner.eu, ...

• [ ] Implement MailSender-Pool which has multiple instances of different Email Service providers, each one with a daily limit and a monthly limit and activation flag configured.

• [ ] MailSenderFactory (and/or MailSender) must be enhanced with the following logic:

• [ ] select Message Jobs in current month => calculate num tasks current day and num tasks current month => return mailsenders which matches Numbers

• [ ] Choose randomly one mailsender

• [x] implement APIs of Email Service and send it

• [ ] persist Mail Sender Pool identity in Messagejob (identify e.g by from-mail config)

• [ ] add configs for mail sender pools (identify by from), add however also service type, Like AWS ses

• [ ] implement bounces handling specific for service provider

• [x] Test has only in-memory Provider, all others are disabled

[danleh]

Hi Clemens! Some of the mails of Mainz kocht, organized via runyourdinner.eu were not send since yesterday. We planned on sending out the routes tomorrow (for about 400 people), do you think the Sendgrid account could be opened up again until then? We would be happy to jump in with some funding, if a "pro" subscription or something of that sort is required. Best and thanks!

[danleh]

If you know the precise time when the account was deactivated, that could help us as well, because we don't exactly know who already got their team assignment email and who didn't. We sent out the largest batch of 132 messages on 2024-11-10 20:39:30. At least some of them (although we don't know how many) have been successfully received by the participants. Another attempt to send out 2 messages on 2024-11-10 23:15:36 seems to have not gone through any longer (the recipients didn't receive any mails, although runyourdinner.eu says "Versand erfolgreich")

[danleh]

We really hope the messages of our event didn't run into some limit of Sendgrid 😟 Fingers crossed we can solve this, let us know if we can help there.

[Clemens85]

Hi Daniel, thanks! Indeed I am thinking about something like a pro subscription, in particular because of a better sender reputation. This is however compared to the other hosting costs quite expensive, and for paying this there would be needed more constant donations.... Hence I am thinking more about a small payment for sending the final routes in future, but this now not the subject. The current issue might have happened also with a pro subscription, the only thing that might differ with a paid subscription is the reaction time of sendgrid, which is however also only an assumption.. Speaking about tomorrow, it all depends on the sendgrid support..I wrote them several messages, but there was no reaction so gar, maybe they will react till tomorrow, maybe not... I encountered a similar issue once in a company I worked for with Mailchimp..hence the motivation to include more than just one mail Provider... Unfortunately this takes time, even when I tomorrow integrated e.g. AWS SES it would take 1-2 days till it would really be ready to use (Domain Verification etc takes time...). Mailing ist just shit... I can try to integrate my runyourdinner Gmail Account temporarily for tomorrow, but it will soon run in a rate limit (i think it is 500 messages / day).

[Clemens85]

We really hope the messages of our event didn't run into some limit of Sendgrid 😟 Fingers crossed we can solve this, let us know if we can help there.

No it was not your messages. I cannot really tell you the exact time, I was notified yesterday at around 4 pm in the afternoon about the account deactivation. So I can for Sure say that any messages after this time were not sent. Unfortunately the Sendgrid API seems to respond in such a case with success... I can also Imagine that this might be intended (maybe they Queue the Messages and send them If it ist activated again..)

[danleh]

Thanks for your speedy replies, as always! :) Let's chat tomorrow afternoon and see if Sendgrid support replies until then.

It would already help us a lot if we can export emails in a csv or json file, then I can prepare it for mailmerge locally with a script and try to send some out on our end (we have our own domain+mail hosting, some rate limit applies, but that's my job to take care of ;) )

When you say gmail account, you mean that would come from a gmail domain, i.e., not runyourdinner.eu, correct? If all else breaks, we would happily take it. Otherwise, export of messages+mailmerge+we send it manually is a fine stopgap. It doesn't need a UI, happy to take some REST API endpoints for creating the emails as well.

[danleh]

Regarding pro subscription for Sendgrid: Do you have an email for me to continue our conversation regarding funding this? We would be happy to wire let's say 50€, possibly more (I'll check back with the team).

[Clemens85]

Hi Daniel,

regarding the funding, there is already a donation button on the web app (PayPal), so feel free to donate ;-)

Regarding the emails: I concentrate for now on bringing up a new email service, I already properly setup domain verification stuff for AWS SES and have a request pending for enabling productive access there. Maybe (with a little luck) it will be processed till this evening, and then I will integrate this instead of SendGrid. Probably I will today also temporarily integrate my Gmail account, but this won't help you, because your message volume is too big.

You can indeed use the REST API endpoints for getting the messages (I won't be able to implement a CSV export in time, unfortunately I am also sick now... sorry for that). The information about the REST endpoints:

A) PUT https://runyourdinner.eu/rest/messageservice/v1/runningdinner/__ADMINID__/mails/dinnerroute/preview (ADMINID is this cryptic string in the URL after /admin/)

You will need to pass a request body like so:

{
  "subject": "",
 "message": "",
 "selfTemplate": "",
 "hostsTemplate": "",
 "teamSelection": "CUSTOM_SELECTION",
 "customSelectedTeamIds": ["__TEAMID__"]
}

subject, message, selfTemplate, hostsTemplate should more or less be self-explanatory. Here you enter the texts in the Web UI (you can also use the template placeholders).

TEAMID must be the ID of the team for which you want to see the preview (this is what you see on the right in Web UI in the preview where you can select each team).

B) To get a list with all your teams you do the following (before invoking the endpoint above): GET https://runyourdinner/rest/teamservice/v1/runningdinner/__ADMINID__?filterCancelledTeams=true There you get the list with teams (and take of course the "id"-attribute). So you can generate the messages for each team (you get a response with two messages, for each participant).

I see there is however one gotcha with the preview response: You get the two messages with the same particpantId, so you cannot match this id with the teamMember Ids in the team-list to figure out the destination mail address. As a workaround you could however take firstname and/or lastname to match them (they will very likele be unique within a team).

Sounds complicated? I would suggest to just try it out, starting with teams-endpoint, and then testing the preview endpoint with one team-id and then you will probably see.

[Clemens85]

Hi Daniel,

one further notice: I now successfully activated another Mail-Service, which is currently in usage. This service has however a rate limit of 200 mails / day. So I would really advice to not use the mailing functions currently as you will run then into the rate limit and furthermore other users are then also locked out. This is just like a first aid for having general email functionality available again...

I got one reply from SendGrid that they might soon reactivate the account again... we will see.... AWS SES is still in review of the AWS team for production enablement.

[Clemens85]

Ok and now a final notice: It seems like that I got AWS SES to work and that you should now be able to send your dinner routes without any issues. This is however quite brand new, so maybe (if you do it) you look carefully about whether your participants got the messages.

[danleh]

(Sorry for the late reply, just came back from work.)

Thanks a ton, this is really amazing work! We will start sending out only a couple mails to people we know and check whether they are received.

And no worries about the CSV export, having the API endpoints is already 95% as good and a great backup, in case the mails via AWS SES end up not working. We will keep you posted, but thanks again for your super quick help and all the effort. Also, please get well soon.

[danleh]

Just FYI: it seems that SendGrid reinstated your account, and all the previously not sent out mails (which were queued up) are being sent out now.

[Clemens85]

Yes I already saw this, the behaviour is like I thought, that they queue it instead of sending an error response. Makes sense, but it doesn't make sense If the account is locked for several days.. I will reactivate the Sendgrid Integration tomorrow, but I will also implement a logic for dealing with multiple Email providers to be more independent in future..

[danleh]

Sure, no haste! Get some rest :) The AWS SES mail integration seems to have worked by the way! We successfully sent about 180 mails this evening, and verified for about 10 that they came through. Thanks again for all the effort.