search

MozillaFoundation / plan

What the MoFo production teams are working on
https://build.mozillafoundation.org
18 stars 4 forks source link

Improve email communication(s) after sign-up #242

Closed thisandagain closed 9 years ago

thisandagain commented 9 years ago

RACI

Our communication after sign-up should draw users back into the product and ideally come after the user has stopped engaging with Webmaker (either via a standard delay or dynamically). In addition, we need to start building up some expertise in what works / what doesn't in regards to email messaging.

Current post sign-up email:

Hi home61,

Welcome to Mozilla Webmaker! Thank you for joining us to explore, play and create together to build a web that's open and made by everyone.

You can get started right now with these free tools:

X-Ray Goggles allow you to see the code behind any webpage and remix elements with a single click, swapping in your own text and images.
Thimble, an easy-to-use HTML and CSS editor, lets you learn code by writing it directly in your browser to create your own webpage.
Popcorn Maker helps you combine content from across the web like GIFs, music, photos and videos into cool mashups that you can annotate and share.
I'm so glad you've joined the Webmaker community. If you have questions, feel free to email me or tweet @webmaker. I'm looking forward to seeing what you make on the web!

Cheers,
Amira
Webmaker Community Manager

If you do not wish to receive any more emails from Mozilla Webmaker you can change your email preferences.

Audience

Newly registered users of webmaker.org

Success

Primary:

Secondary:

As a baseline I'd like to see some updates to the post sign-up template (text-only / above) but ideally we could get a more robust system in-place that would allow for better tracking of open / click-through rates as well as allow for more control of timing.

thisandagain commented 9 years ago

One technology that I've used in the past as to make the templates more accessible as well as easy to track is http://apostle.io/

ScottDowne commented 9 years ago

Is there something I can start on with this? Not really sure what's needed yet.

thisandagain commented 9 years ago

@ScottDowne @HPaulJohnson Can you two sync-up?

ScottDowne commented 9 years ago

We are, but I was hoping there was something that could be dropped in the ticket vs vidyo?

thisandagain commented 9 years ago

So I think the big question for me is if we can replace / augment our existing transactional email system with something that allows for:

Apostle is one such option that I'd vouch for, but we could also get some of these features out of other tools like Mandrill. With any option, some documentation dropped into this issue about how our current system is architected and your opinions on 3rd party options would be really helpful.

HPaulJohnson commented 9 years ago

@ScottDowne and I met this afternoon. Next steps:

Daily standup scheduled between now and end of sprint for 11:30 am PT in my vidyo.

simonwex commented 9 years ago

@adamlofting do you know how many emails we send per month?

@jbuck do you have an overview diagram including sawmill, webmaker-mailroom & the templates?

simonwex commented 9 years ago

@ScottDowne feel free to book time with me if you want to brainstorm.

adamlofting commented 9 years ago

As far as I know our current transactional emails don't produce any data (or if they do, I just haven't seen it). If there is any data, I'd love to look at it.

I've used Mandrill in the past and the analytics was good, but I had to build a custom editor for the content. It looks like Apostle saves us from doing that, and having A/B testing built in is excellent :metal:.

I'd like to add a requirement to this task that:

jbuck commented 9 years ago

@simonwex Nope, no diagrams. But conceptually it'll be really easy to use apostle.io in place of webmaker-mailroom

thisandagain commented 9 years ago

@adamlofting @simonwex Amazon SES kicks out some basic metrics via CloudWatch. We might be able to get a sense for monthly volume from that.

ScottDowne commented 9 years ago

Apostle demo: https://github.com/mozilla/sawmill/pull/35/files

To test I have been doing this:

Create a file called test.js

Content of file are:

var worker = require("./worker");
var apostle = require('apostle.io');
apostle.domainKey = "111c1d90d76663230994fa33e64b068ae11c5302";

worker.send_new_user_email(apostle)(null, {
  event_type: "create_user",
  data: {
    email: "something.something@gmail.com",
    username: "Joe Son"
  }
}, function(err) {
  if (err) {
    console.log(err);
    return;
  }
  console.log("email sent");
});

Then run node test.js

simonwex commented 9 years ago

We don't have any SES stats in CloudWatch.

@jdotpz doable to figure out how many emails we send a month on average?

jdotpz commented 9 years ago

Sure, I'm betting our billing info will have it. I'll look it up tonight or in the am and shoot it over.

jdotpz commented 9 years ago

TLDR: For the last 3 months of 2014, we averaged 1029 emails per day, or 31584 emails per month. That said, we're already way over that in Jan, where we are averaging 1770 emails per day, or 54889 emails per month.

The data: Date # sent 2014/01/31 23:59:59 10277 2014/02/28 23:59:59 7045 2014/03/31 23:59:59 11969 2014/04/30 23:59:59 9340 2014/05/31 23:59:59 8656 2014/06/30 23:59:59 10293 2014/07/31 23:59:59 15289 2014/08/31 23:59:59 51648 2014/08/31 23:59:59 62000 2014/09/30 23:59:59 33063 2014/10/31 23:59:59 23856 2014/11/30 23:59:59 41363 2014/12/31 23:59:59 29536 2015/01/31 23:59:59 38954

jbuck commented 9 years ago

I suspect a large part of that is that login3 is much less friction to sign in.

On Fri, Jan 23, 2015 at 9:51 AM, JP Schneider notifications@github.com wrote:

TLDR: For the last 3 months of 2014, we averaged 1029 emails per day, or 31584 emails per month. That said, we're already way over that in Jan, where we are averaging 1770 emails per day, or 54889 emails per month.

The data: Date # sent 2014/01/31 23:59:59 10277 2014/02/28 23:59:59 7045 2014/03/31 23:59:59 11969 2014/04/30 23:59:59 9340 2014/05/31 23:59:59 8656 2014/06/30 23:59:59 10293 2014/07/31 23:59:59 15289 2014/08/31 23:59:59 51648 2014/08/31 23:59:59 62000 2014/09/30 23:59:59 33063 2014/10/31 23:59:59 23856 2014/11/30 23:59:59 41363 2014/12/31 23:59:59 29536 2015/01/31 23:59:59 38954

— Reply to this email directly or view it on GitHub https://github.com/MozillaFoundation/plan/issues/242#issuecomment-71203775 .

thisandagain commented 9 years ago

Yeah... new account sign-ups are way up post splash / login changes: https://mozillafoundation.geckoboard.com/dashboards/F62088172D822E2A

simonwex commented 9 years ago

Notes from Friday Jan 23

Action Items:

HPaulJohnson commented 9 years ago

@thisandagain, question about content of an updated email:

In the new version of the email where should we point people in our CTA? A specific x-ray goggles activity is my preference but we can test to optimize engagement/retention. Other options - given current site functionality - would be tools, explore or profile.

I'll work around this for now as I update copy but resolution would be good early next week.

simonwex commented 9 years ago

I'm moving this to P2, @ScottDowne is going to be focusing on #263 (timing-critical) next week and getting the planning work done for this as his time permits.

ScottDowne commented 9 years ago

@HPaulJohnson @thisandagain @simonwex I heard back from Apostle.io about privacy. There is currently no way to use the email authoring UI or AB testing without sending the user emails to a third party.

What does this mean for us?

davidascher commented 9 years ago

We either need to do a legal vendor review of apostle.io or write something similar ourselves, methinks.

simonwex commented 9 years ago

@thisandagain introduced us to Mal, the founder. I'll email him and see if they could prioritize a UUID-based call, in conjunction we should kick-off a review. @ScottDowne let's do that "paperwork" together today.

simonwex commented 9 years ago

Emailed Mal and CC'd scott.

OpenMatt commented 9 years ago
HPaulJohnson commented 9 years ago

This was bumped to next heartbeat as devs were re-assigned to Net Neutrality call tool project. On today's TPS call we can decide if that's far enough along to restart work.

simonwex commented 9 years ago

Let's chat about this during our TPS on Monday. It's likely going to come down to this or #160.

thisandagain commented 9 years ago

Moving #160 to the backlog.

adamlofting commented 9 years ago

On a related note, this new dashboard shows the number of new users p/day who opt-in to receive email communications (it's ~50% of sign-ups):

I suspect the remainder of the daily emails shown in the stats above relate to things like login emails / reminders etc.

thisandagain commented 9 years ago

I propose this be a P2 for this sprint given the technical debt reduction and parallel login / sign-up work.

OpenMatt commented 9 years ago
OpenMatt commented 9 years ago

(will leave this as an open planning ticket for now -- @HPaulJohnson please close and file as needed once the kick-off happens / next steps get logged, etc.)

HPaulJohnson commented 9 years ago

Summary for easy reading and as we explore whether this ticket can be rolled into #53.

We need a better solution to support transactional/triggered emails from webmaker.org and teach site. The welcome email triggered at login is the first example of this but there can/will be many more instances.

The key requirements are:

In the first go around with this ticket we looked at a small number of 3rd party solutions, giving the most attention to Apostle.io. Apostle was tested by Scott and Simon had begun exploring legal/approval process for agreement before this ticket was reprioritized in favor of Net Neutrality work that was more time sensitive.

@secretrobotron to decide whether we close this ticket and move to #53 or keep this part here.

secretrobotron commented 9 years ago

:+1: Confirmed with @ScottDowne and @simonwex:

secretrobotron commented 9 years ago

@simonwex any news on Apostle review?

ScottDowne commented 9 years ago

@secretrobotron I'm submitting it. I had some questions about @simonwex 's instructions so once I get a few answers I should be good to kick this off. The weekend caused a delay here.

I need to know when we expect this to release. I feel like March 27th (listed in this bug is too soon for the review?) But not sure how long I should expect a review to take.

secretrobotron commented 9 years ago

This ticket will definitely have a follow-up. March 27 isn't an implementation deadline. Let's just make sure the process is started this week.

ScottDowne commented 9 years ago

The process to start is a form that has a due date as a required field.

I need to put something in to start the process.

secretrobotron commented 9 years ago

Ah. @HPaulJohnson or @hannahkane might have an opinion. Would be nice to have the review done by March 27, but I'm not sure how long they might take. @simonwex might have some idea though.

davidascher commented 9 years ago

Just say ASAP / March 27 ​

OpenMatt commented 9 years ago

status?

OpenMatt commented 9 years ago

@ScottDowne is doing vendor review. He will update this ticket this morning.

ScottDowne commented 9 years ago

@OpenMatt nothing has happened in the security review bug, but the legal review bug has some questions for us/apostle that I need to see how many I can answer and how many I need to push to other people.

(1) What do we intend to use these emails for? (2) They suggested ExactTarget as an alternative to Apostle. (3) Linked us this: https://www.mozilla.org/en-US/privacy/websites/ calling out the ability to let users view text only emails. (4) What analytics data do they receive from emails? (5) What data to the collect and store, other than email address? (6) In what country is this data stored? For how long is it stored? (7) Can it be deleted at the user's request? Or Mozilla's request?

simonwex commented 9 years ago

Hey @ScottDowne, 1) is covered above, 2) we should talk about and the rest you should get into contact Mal about.

On ExactTarget v.s. Apostle, I believe this is worth a discussion. It is a salesforce company, so integrations with our CRM is likely. MoCo is already making use of it, as are we in some cases.

@thisandagain, I think ExactTarget would suit this use-case, do you agree?

@davidascher do you know how we can figure out what pricing would be for ExactTarget for us?

valianttry commented 9 years ago

https://github.com/MozillaFoundation/plan/issues/346#issuecomment-87826672

I strongly urge us to implement ExactTarget for all Mofo's email marketing -- I thought that was the intention (but clearly there's a disconnect somewhere). Since ExactTarget is already through legal and security review, we already have a signed contract and can begin implementation -- and based on my understanding of Webmaker's transactional email strategy -- it should meet the needs.

valianttry commented 9 years ago

Here's a blog post about ExactTarget, Journey Builder, and Transaction Builder (all part of the Exact Target suite): http://blogs.perficient.com/digitaltransformation/2013/11/18/dreamforce-13-email-marketing-with-exacttarget-marketing-cloud/

"Transaction Builder is used when you wanted to send triggered and transactional emails. For example, when a customer interacts with you, you want to send them emails based on actions they take. For example, when you rent a movie at Redbox, you receive an email as soon as you get the movie from the machine. ExactTarget sends millions of these transactional emails for Redbox. Triggered and Transactional emails are the most opened emails. ExactTarget uses a robust API to allow your internal systems to connect to the transaction builder. ... This includes dynamic content, testing & optimization, and mobile content for email and mobile messages. The content system is built on an open platform that allows you to create “gears” that can be included in the mail. Nike used ExactTarget to go from 2 customized emails (men vs women) to over 1.3 million versions without additional staff. Content can be rendered at send time or at run time. Run time content can be customized for the user every time the email is opened. "

thisandagain commented 9 years ago

It sounds like this may cover our use case. I'd love to get into the details as I'm having trouble finding documentation, but if we can use ExactTarget I'd much prefer to have our transactional and marketing emails together within the same system (AND avoid yet another vendor review). Thanks for the catch @valianttry !

Will schedule a follow-up so we can dig in.

valianttry commented 9 years ago

@thisandagain sounds brilliant -- also I want to talk to you about localization management... want me to schedule 30m for you and I for these 2 things? Who else should join us?

HPaulJohnson commented 9 years ago

I would like to join!

thisandagain commented 9 years ago

:+1: