Design Sequence Messaging

[jekram]

Design Sequence Messaging.

As part of the design of the sequence messaging, first, investigate how sequence email is implemented. Define the core features and full feature list Sequence Messaging. As part of this design also identify the reporting needs.

[sojharo]

Steps:

1. Learn what email sequencing is
2. ManyChat implements similar concept for message sequences which are basically automated messages scheduled.
3. Design the similar logic for us.

[jekram]

@AnishaChhatwani

Please read this: https://webprofits.agency/blog/ultimate-guide-email-sequences/ It is a good summary of email sequencing

@sojharo @dayemsiddiqui @ImranBinShoukat

[AnishaChhatwani]

https://docs.google.com/document/d/1C4OV-xUZ1PxKaVvf4DsMDpCqHxh21u3UcGDpWn8pn70/edit#heading=h.dj6z7z2d9anf @sojharo @jekram please review

[sojharo]

@AnishaChhatwani I have read it. Please let me think more on this. I think it would be better, if we run separate micro service for this purpose which would just manage the scheduling of messages. It would need to be handling lots of scheduled messages at a time and will be responsible to send the scheduled messages or let the host server cancel them. I would think more on this as soon as I am done with marketing and revamping stuff.

[jekram]

@AnishaChhatwani @sojharo 1. How are we setting the timer and coming back 2 days later? 2) We need a more detailed logic flow

[sojharo]

The cron job library for node.js automatically handles this for us. We would give it the date on which we want to schedule some event. When the date occurs the callback is called to carry on the event. Library will call that event 2 days later. The scheduler logic is handled by cron job library.

[jekram]

Hum.... Is that scalable? Still would to see the detailed logic flow

[sojharo]

Yes, this is scalable and also to make it easy I am thinking to run this as separate micro service. I would create the detailed logic flow.

[sojharo]

https://docs.google.com/document/d/1C4OV-xUZ1PxKaVvf4DsMDpCqHxh21u3UcGDpWn8pn70/edit#

Let's discuss this once we are done moving to Azure.

[sojharo]

Following is summary of microservice structure:

[jekram]

@sojharo @AnishaChhatwani

From a coding perspective, we should write as separate logical structures., so we can later split them apart.

[jekram]

@AnishaChhatwani This document lacks what we are doing from a functional perspective. Have you read how sequencing is implemented in an email? Would like to understand how does it compare for example to SendGrid or MailChimp? What functions they are providing and what functions we are providing or not providing. It needs to end to end. From setup to reporting.

[sojharo]

Let me look into SendGrid and MailChimp to see how email sequencing is done. The current document is looking at how ManyChat is doing sequencing. I will see how MailChimp or SendGrid are doing this. I had read one article on email sequence that you had given me. I will do the comparison in this document today after my current issues. From ManyChat, we are providing all the functions.

[sojharo]

SendGrid is calling it a Drip Campaign and they don't have such features in their dashboard for email sequencing. Our server will be responsible to call the API to send the email sequencing emails at the right time. Here is their article:

https://sendgrid.com/docs/Glossary/drip_campaign.html

Mailchimp however has this option in dashboard, they call it Marketing Automation. I couldn't test it but I have read the documentation.

https://mailchimp.com/resources/guides/getting-started-with-marketing-automation/

They are not different from the article that you had given me to read about email sequencing.

I went through these articles and then our design document. I need to modify the document for some corrections and improvements that I would do today.

[jekram]

Most of the people are using MailChip for Marketing Sequence. Most of the FaceBook article refers to integration with MailChimp. I will get MailChimp and we should model our product with MailChimp

[sojharo]

Today, I am going to watch some "email sequencing with mailchimp" tutorial videos to understand the model of mail chimp. I wasn't able to sign in with credentials and also I saw some articles regarding email automation on mailchimp website. These articles were more like marketing material and didn't explain much about how mailchimp has done sequencing.

I will start from this video.

https://www.youtube.com/watch?v=tGVaL8pOdH8

[sojharo]

From following video about MailChimp, I learnt that we should combine our message sequencing with our welcome message so that admin will easily know the flow of message series. For example, admin will know that on first message subscriber will receive this welcome message and then it will receive this message after 2 days, and then this after 5 days etc. This would be easier from user perspective

https://www.youtube.com/watch?v=tGVaL8pOdH8

Here are some things that we need to learn about MailChimp first and then slowly move to it:

1. New Signup should add that email to our MailChimp list with Welcome Email campaign

So, whenever user signs up on KiboPush, we should give his/her email adress to MailChimp via API. This email address will be added to let's say welcome email list. He would immediately be sent welcome email by Mail Chimp. So, instead of us writing the logic to send email on server (that we currently do) we will have a designed template of email on mailchimp tied to welcome email list campaign. I have seen how we can add the email to a list on mailchimp:

http://developer.mailchimp.com/documentation/mailchimp/guides/manage-subscribers-with-the-mailchimp-api/

2. Utilize power of multiple lists

We should have multiple contact lists on MailChimp, let's say, we have one list for those who have already signed up, and those who are our target audience. Both lists will receive different emails. Once the person signs up, we would do the API call to move that person from target audience list to signed up list.

3. Campaigns can be tuned to make email sequencing

Campaign starts with one email and then we can add further emails to it for automating the email series. So that they would go one by one on defined intervals. We need to setup campaigns entirely on mailchimp and not on KiboPush for our marketing.

4. Emails with data

Only challenge I have seen in mailchimp is how to send email with data. For example, our account verification email contains the verification link so data is involved here. We can't predefined email for this purpose. Another case is where we send weekly email. So, I haven't find any API in their reference with which we can put data into predefined email or send email from our server (just like we do with send grid). For this, mailchimp gives us a sister site called ManGrid ( http://mandrill.com/ ) . I am still looking into oAuth API of mailchimp, this might give us any solution for sending emails with data.

In sendgrid, I can define the body of email on my server and then call the API to send it. In MailChimp, I can add the email address to a list and tie that list to a campaign. Campaign will contain emails to be sent. So this is a difference in Sendgrid and mailchimp. I think the ManGrid of MailChimp is a better solution where we will not have to define the email body on our server and body will be defined on ManGrid and our server will just send the data that needs to be put in that body.

In this, I need to explore more. I am going to ask question from MailChimp support to understand how we can send email which contains dynamic data.

We can easily start mailchimp with points 1, 2 and 3. For point 4, I need to research more and for now we would be doing it with sendgrid.

As far as design of Message Sequencing is concerned, I think I have learnt few things in videos and articles about email sequencing and once we use MailChimp for our own email automation, we would understand more deeply. So I think for two weeks, we can defer its design process until by using we grasp all the technicalities. However, I have got enough knowledge that I can continue the design of Message Sequencing in this issue. I just want to be little careful in implementing its design. Because when we use email sequencing more and more, we would learn more about it and our design would be better and costumer centric.

[jekram]

@sojharo Great analysis

Please open task KiboPush implementation of Sequencing

[sojharo]

I looked into the technology for scheduling today. We have two following options:

1. We run microservice with nodejs code to do scheduling using library e.g. node-scheduler or later
2. We let the cloud scheduling services handle the scheduling logic for us

I have detailed about procedures:

1. Node.js cron job library: node-scheduler or later

With this, we would have to write nodejs service which would be written with help of cron-job library. We would write the logic to how to start cronjob or cancel it. We would have to identify for which company and page this logic is running. Following are some node-js scheduling libraries that I have seen and understood their implementation today:

https://www.npmjs.com/package/node-schedule

https://bunkat.github.io/later/

However, most of these are not persistent. We would have to write our own logic to make them persistent by writing them to mongodb.

2. Cloud Scheduling services

Other option for us is to let the cloud handle the procedure and logic of scheduling. I have seen two cloud scheduler services. One of them is from Azure. In this, we would define the tasks that could be scheduled in cloud. i.e. send this message to this subscriber after 2 days. We would also give the webhook to cloud schedular. When it is time for the event, the cloud schedular will call our server webhook. Our server would then send the message which was scheduled for that subscriber.

With this, we would not have to maintain a separate microservice. We would have schedular running in cloud and it would keep telling us when the scheduled job is read to run. This would help us in both scheduling broadcasts and message sequencing.

Here are following cloud services:

https://blog.iron.io/how-small-nodejs-project-turned-into/

https://azure.microsoft.com/en-us/services/scheduler/

From carefully reading about documentation of both services, I think Azure would be more helpful for us as it has monthly cost of $13 as basic plan. It has very good API to schedule and cancel jobs. I think we would be good in going for Azure if we decide to use cloud services.

[sojharo]

I adjusted the design document for how we should do it for Azure Scheduler. I have also changed some tasks and database design. This is complete now. @AnishaChhatwani can open the tasks and she can discuss the document with me once if she feels need.

I have added one task to understand Azure Scheduler.

[jekram]

I am closing this task

[jekram]

@sojharo Please update the design document to reflect above

[sojharo]

https://azure.microsoft.com/en-us/services/scheduler/

We should use this service. Kindly let me know which thing should I reflect.

[ImranBinShoukat]

I have gone through the design document and understood it. I studied the ManyChat sequence messaging also.

I found that the tasks list is not complete. There should be tasks opened to cater these:

1. UI and Logic to display all the sequences
2. UI and logic to update sequence
3. UI and logic to update messages
4. UI and logic to view messages
5. Mock up for create, update, and display all sequences
6. Mock up for create, update, and view messages
7. There are no tasks for sending socket.io updates
8. To cater welcome message in sequences what we can do is that we can give users a default sequence (i.e Welcome Message Sequence) and user will only be able to update it or enable/disable it.

Should I open these tasks?

@jekram @sojharo @AnishaChhatwani FYI

[sojharo]

Yes please go ahead and open the tasks

[ImranBinShoukat]

I have modified the design document and added some changes.

@jekram @sojharo @AnishaChhatwani please review the changes as these are very important changes and it changes the overall structure of sequence messaging.

Also, I have opened the tasks as well.

[sojharo]

@ImranBinShoukat in this task please do the design for those features of message sequencing which are in MailChimp i.e. sequencing based on actions.

[ImranBinShoukat]

@sojharo @jekram we can close this task. We are already discussing the v2 design document in #2643.

[jekram]

Can we discuss this in the meeting today