Update Scheduled Notifications to Perform at Scale

[Davinkjohnson]

@we-ai had some ideas on how this might be handled.

Some ideas were around memory size, async threads, query optimization.

[we-ai]

Previous updates on scheduled notifications:

• Issue 692: >200x speed-up handling notifications
• Issue 827: Send SMS messages together with emails

With above updates, currently in a scheduled run, the system takes about 320 seconds to send out 16k emails and 16k SMS messages, but this isn't enough because the number of participants grows rapidly everyday. Now we have close to 40k participants. With timeout restriction of 9 minutes (540 seconds) on the notification handler in GCP, it cannot send out emails and SMS messages to all (close to 40k) of our participants in one run.

This is why our scheduled notification system needs updates. Ideally, it can efficiently send out +200k emails and SMS messages in a single run.

[we-ai]

For SMS messages in scheduled notifications, do we need to include personalized info (first name, token, specific links, etc)? @HanaShiho @brotzmanmj Personalized messages can only be sent one by one in Twilio, while generic messages can be sent out in bulk which is faster.

[brotzmanmj]

Hi, I defer to @HanaShiho as to whether this is a necessity to avoid our messages getting labelled as junk, etc. I recall the Chicago team mentioning that personalization of the messages was important but don't know if we decided that was relevant to us or not.

[HanaShiho]

@we-ai @brotzmanmj So this is an interesting one. In some cases personalizing the messages can help against being flagged as spam in SMS just as they appear less "bulk message" focused, but the nuance was that if we aren't already being flagged as spam for SMS (we haven't had any yet) then we probably don't need to make the change. It would also increase character count in a segment or dilute how much space left to write the msg.

At a high level, I'd say unless we think we have a problem being filtered at the moment, the personalization may not warrant the effort. It works great in email as they tend to be longer msgs and it encourages engagement. However, with our use case, I think as long as the recipient immediately knows its from Connect (i.e. they know they are in the study so the msg is relevant) then that may be enough.

[we-ai]

@HanaShiho Thanks for your comments. So bulk messages are fine for us. I'm checking sending SMS messages in bulk and will update here after I have more details.