[Tracking] Add email tracking for email deliverability and open/click rates.

danielrvidal commented 2 months ago

Proposal: Add email tracking for email deliverability and open/click rates.

Problem: Recipients are not reliably clicking on IOUs from senders, and thus never coming into the product. We know a non zero percentage of users ignore the emails we send, but we don’t actually know how bad the problem is. We rely on email notifications to pull viral leads into our product. However, we’re unable to determine if our emails are making this problem bigger by not being engaging and compelling for our users, or even worse not getting delivered. This lack of insight prevents us from optimizing our email communications, resulting in decreased user engagement, decreased understanding of our value, decreased conversion rates, and ultimately hampers our ability to drive sustainable growth.

Solution: Let’s start tracking email deliverability and open/click through rates with all relevant notifications so we know how big the problem is and can a/b test email communications to increase conversion of invited users to active users.

Let’s start by tracking the emails we send for one invite type: IOUs for new users.
We already use custom subject line to trigger an immediate unread message notification to the IOU recipient (added in this PR). Let’s add a tracker in this same place that identifies the type of email.
- It might look something like $this->parameters['trackingKey'] = 'IOUnewUser'
We’ll send this tracker as a tag to Mailgun
- Here is an example of where we’ll put it.
Mailgun is already set up to measure email deliverability and open/click-through rates for all emails. All we need to do now is distinguish an IOU notification vs. any other unread message notification and the tracker accomplishes that
- Note: Mailgun only covers 50% of our emails. But so long as we track the control and test versions and compare them within that 50% pool, and then wait for statistical significance, we should learn if/which notifications are effective.
We’ll run an inbox placement test which sends the IOU notification to various mail providers and lets us know how often the specific notification is getting caught by spam.
We will get a baseline on the current notification open/click rate and validate this design before we introduce any A/B testing
- A/b testing the open rate of IOUs for new users using different subject lines will likely be our first test which is why we chose that, but the a/b test will come later.

Tasks

[x] Post Proposal (full Problem/Solution statement) in #expensify-open-source
[x] Wait at least one full business day, and until the post has a majority (2/3) of positive reactions (👍)
[x] Paste Proposal in the space above with a link to the Slack thread
[x] Email strategy@expensify.com and paste in the Proposal
[x] Fill out the High-level overview of the problem, Timeline, and Terminology sections of the Design Doc
[x] Email strategy@expensify.com (continue the same email chain as before) with the link to your Design Doc
[x] Host a pre-design meeting (example) in #expensify-open-source to discuss any necessary details in public before filling out the High-level of proposed solution section.
[x] Fill out the High-level of proposed solution section
[x] Email stategy@expensify.com again with links to the doc and pre-design conversation in Slack
[x] Add the DesignDocReview label to get the High-level of proposed solution section reviewed
[x] Respond to any questions or concerns and bring up blockers in Slack to get a consensus if necessary
[x] Confirm that the doc has the minimum necessary number of reviews before proceeding
[ ] Host another pre-design meeting in #expensify-open-source to ask for engineering feedback on the technical solution.
[x] Fill out the Detailed implementation of the solution and related sections.
[x] Re-add the DesignDocReview label to this issue
[x] Respond to any questions or concerns and bring up blockers in Slack to get consensus if necessary
[x] Confirm that the doc has the minimum necessary number of reviews before proceeding
[x] Email strategy@expensify.com one last time to let them know the Design Doc is moving into the implementation phase
[x] Implement the changes
[x] Add regression tests so that QA can test your feature with every deploy (instructions)
[ ] Send out a follow up email to strategy@expensify.com once everything has been implemented and do a Project Wrap-Up retrospective that provides:
- Summary of what we accomplished with this project
- What went well?
- What could we have done better?
- What did we learn?

Issue Owner

Current Issue Owner: @danielrvidal

danielrvidal commented 2 months ago

Here are the next steps as of now

[x] @justinpersaud - Give access to Mailgun: https://github.com/Expensify/Expensify/issues/426298
[ ] @justinpersaud - Update IOUs to have the tag: https://github.com/Expensify/Expensify/issues/426303
[ ] @justinpersaud - Update Mailgun to store the tags for IOUs: https://github.com/Expensify/Expensify/issues/426306

From there we'll plan the next steps once those are done.

justinpersaud commented 2 months ago

Workong on it this week but I'm just trying to wrap up some SOX audit issues that keep coming back from EY + finishing up blocking spam with ChatGPT for #fireroom-05-23-spam

danielrvidal commented 2 months ago

We should have the doc out today or tomorrow!

justinpersaud commented 2 months ago

I updated the design doc and filled in the Data Storage / API section

danielrvidal commented 2 months ago

High level design doc

The HL is ready for reviews. Please give it a read and ask questions in the doc or room. Assigning the review label now!

melvin-bot[bot] commented 2 months ago

:wave: Hello Generalist Track Team - you have been assigned to review this High Level Design Doc. Check out this Stack Overflow for some tips on reviewing a design doc. Once you are done, simply press the Add "Reviewed Doc" comment button in the right hand side K2 panel or follow these instructions.

@srikarparsi (Expensifier / Graduate) - https://github.com/Expensify/App/issues/49304
@Christinadobrzyn (Expensifier / Graduate) - https://github.com/Expensify/App/issues/49305
@zsgreenwald (Project Manager) - https://github.com/Expensify/App/issues/49306
@tylerkaraszewski (Project Manager) - https://github.com/Expensify/App/issues/49307
@twisterdotcom (Product Manager) - https://github.com/Expensify/App/issues/49308
@stephanieelliott (Product Manager) - https://github.com/Expensify/App/issues/49309
@davidcardoza (Generalist) - https://github.com/Expensify/App/issues/49310
@garrettmknight (Generalist) - https://github.com/Expensify/App/issues/49311
@MarcFillari (Accounting Technical Team) - https://github.com/Expensify/App/issues/49312
@CherylWalsh (Marketing Team) - https://github.com/Expensify/App/issues/49313