Messaging: Onboarding 5.0

[joethreepwood]

Context

Part of https://github.com/PostHog/meta/issues/171 Follows https://github.com/PostHog/meta/pull/150

Apologies for the readability of this. If you feel there's a better way to review this, or if you'd like to hop on a call to feedback, LMK!

Concept

Onboarding 4 focused on introducing upsell content, to get people paying for each feature.

Now we want to focus more on activation. Get them using us. Then upsell.

At the same time, we also want to personalize the emails more to help people get started.

And we're introducing a revised in-app onboarding that lets users choose a single product to get started with. If we can harness an event from that, we can tailor email onboarding to focus on that firstly and then take them through a flow tailored to them - with subsequent upsells and emails about how products combine.

Implementation

This flow is exponentially more complex than the previous flow. Therefore, the current version just includes logic. I want to get feedback on logic first, before I write dozens of emails.

Here's the high level view of what this flow does that's new:

• It checks which product users selected first in the in-app onboarding, as an intent signal
• It waits a few days, then checks if they have activated with that product.
• If they have activated, we send them an email inviting them to sign up to the community.
• If they haven't, we send them an email to help them activate.
• A week later we check if they have subscribed to their initial product.
• If they have, we do nothing.
• If they haven't, we send them an upsell email.
• We then wait until they have activated in ANY product. If they have already activated, they wait 1 week.
• We move through the remaining products and send them the upsell emails, with content that will be tailored to combining with their initial product.
• At the end we invite ALL users to view betas

If users don't select an initial product they move through a condensed version of the above without the first activation check.

Here's what the full flow looks like.

Requested feedback

Everyone has opinions on email content and we are not at that point yet. This is purely to figure out what's practical, and logical. As such I'm looking specifically for feedback from @raquelmsmith @xrdt and @charlescook-ph on a handful of points:

• What do we consider the activation criteria for each product? We will use this as an email trigger.
• Will we add events to the in-app onboarding reporting to Customer.io which product was selected for first onboarding? What are they?
• Do we feel this level of emails is excessive, or do we feel it is acceptable? I'm tempted to space it out further, but this will mean users could take 2+ months to complete the flow. Alternatively, we could remove upsell/activation emails from each 'track' after the first set is sent.

[joethreepwood]

Had a chat with Charles today and have redone the flow with some changes. I've updated the post above, and added a quick explanation of the concept.

Changes are that I've removed the repeated activation steps for each product, so that we now only check for activation of the initial product. I've also introduced a fall-back email for the first step, a final email for the end of the flow, lengthened some time delays and added a new step so we only try to upsell users who have successfully activated.

@raquelmsmith let me know if you have any strong thoughts here, otherwise I'll start drafting content and adding detail soon.

[raquelmsmith]

This is great! I'm excited to get more activation-focused stuff in there 🤓

For the activation emails, IMO that should be the very first thing we send after the welcome email. Currently it waits 5-7 days to send the activation email, and by then it may be too late, they've moved on to another company's product.

Before building this all out I would be (apologetically) curious about our current conversion rates on the types of emails that are being proposed here. Types I see are:

1. Welcome email - is this useful? Do people open and click?
2. Newsletter invite - does it work? Should it be combined with the community invite so everyone gets that one too?
3. Advice for X teams - is this new? What do we want people to do with it?
4. Community invite
5. Activation help emails - running a couple experiments here, one seems to do something, the others don't. Curious if you have more data.
6. Upsell emails - do these actually work?
7. Betas invite

What is the reasoning behind inviting people to try out betas? That doesn't seem like a super necessary email in this flow - instead I think it could be a CTA at the end of changelog emails, where it's more relevant if we're announcing a beta for a feature/product.

[joethreepwood]

Short version

Most of the existing content works well. Where it doesn't, the plan is tailor the content to the users more rather than to remove the email. Our open rate across the entire flow is 49%. CTR is 3.5%. Conversion to billing product activated within one week is 1.7%.

Long version

For the activation emails, IMO that should be the very first thing we send after the welcome email. Currently it waits 5-7 days to send the activation email, and by then it may be too late, they've moved on to another company's product.

I softly disagree here because

• Practically the emails which go between the Welcome and Activation emails are focused either on other activities we value (newsletter signups) or helping users understand important concepts (actions). The newsletter invite currently performs well (below).
• Conceptually I think the in-app onboarding should be how we focus on driving immediate activation, and that emails strength is that we can follow up a little later. 5 days feels about the right amount of time for such a follow-up without sending too many emails.

How is Onboarding 4.0 performing?

It will be swayed somewhat by the fact that Onboarding 4.0 is still relatively new. Take the below with a small pinch of salt due to Xmas.

All these figures are avgs over the last 30 days unless specified. As a baseline I'd normally expect a 40-45% open rate and a 3-5% CTR.

Welcome email

Open rate: 60% avg CTR: 8% avg

Newsletter invite

Open rate: 64% avg CTR: Varies depending on founder (approx 20%) and engineer (approx 5%)

This gets us approx 75 clicks/subscriptions to Substack per week. I can figure out the full subscription rate here, but it's error-prone and a bit of work.

I don't think we should merge this with the community invite, as the newsletter and community exist in different places and for different audiences.

Advice emails

These were new in Onboarding 4.0. We felt non-technical users would greatly benefit if they knew how to create actions without code. If they have created an action they get a general advice email. If they haven't, they get advice on doing that.

Sample sizes skew things a lot here. Open rates average around 50%:

CTR is around 3%.

So, we can probably optimize this content, but the open rate seems solid.

Community & Beta Invites

These are currently only sent as fallback emails. If a user is subscribed to every single product, only then do they get these. In Onboarding 5 I've added them as best practices we want people to be aware of and to help break up the emails away from being salesy. We mention the betas in changelog when relevant, but I think a general encouragement here is good too but it's not a strong feeling.

Community invite Open rate: 55% CTR: 4%

Beta invite Open rate: 58% CTR: 6.7%

I feel like this is pretty solid for now, and it'll likely improve if we move it to not be a fallback.

Upsell emails

There's a lot of these, so this is quite involved. The short version across all of them is:

Open rate: Approx 45% CTR: Approx 1.5% Conversion to billing product activated (below) is around 1.5% too.

So, these aren't performing particularly well. The hope with Onboarding 5 is that we'd tailor them a bit more based on the first selected product, so that's the plan for improving these.

[joethreepwood]

I've now plugged in all of the logic for the flow, so next week I'll start looking into content creation a bit more. That's likely to last a sprint or two.

[joethreepwood]

@raquelmsmith as per feedback on the activation tests, I'm going to create a branch for the welcome emails here.

After 1 hour, non-engineers will get this:

While engineers will get this:

[joethreepwood]

I've made some further logic tweaks.

The notable changes:

1. We now send a different welcome email to users depending on if they are an engineer or not. Engineers get given a selection of ways to install and encouraged to setup a reverse proxy. Non-engineers get a link to an AARRR template and encouraged to invite an engineer.
2. I've added a maximum wait on the pause that takes place. It used to be that, if someone never activated, they'd never get upsold to another product. Now, that pause is capped at 6 weeks. After this they get moved along in the flow as a way to re-engage them.
3. At the second upsell point I've added a fall-back email to invite them to betas. Again, this is to help engage them beter and to break up the flow of emails a little.
4. I've added a fall-back email on the first activation check. It used to be that, if a user had activated, we'd say nothing straight away. Now, we tell them about notebooks.
5. I've also started rolling out new designs for the emails, based on ones by Cory.

[joethreepwood]

OK, new and hopefully final version.

• Following conversations with @andyvan-ph and @raquelmsmith I've removed the entire first chunk of the flow. The purpose of this chunk was to encourage non-technical users to discover actions, and to encourage technical users to subscribe to the newsletter. We decided these were lower value than the activation criteria and, since Raquel strongly felt that a successful replay activation test she'd been running benefited from a fast-send, we've prioritised that.
• We now use the newsletter invite and action emails only as fall-backs for the initial activation checks. If a user has activated within one day we invite them to the newsletter. If not, we try to activate them and do not ask them to subscribe to the newsletter at all.
• I've deleted the entire high ICP flow at @camerondeleone's request. We'll pursue a different, separate solution here.
• I've added an additional A/B test building on @raquelmsmith's email test. If someone hasn't ingested recordings they'll be sent one of two emails. One gives them direct information about setting up replay. The other gives them a case study to show the value of setting up replays.
• I've spread other existing emails out throughout the funnel so there is a fall-back for each step, pointing to an additional valuable feature or resource.
• I've added an additional step to the generic flow. If someone doesn't select anything during product onboarding, we now check after 1 day if they've reached the criteria for ANY product. If yes, they get a newsletter invite. If no, they get an actions informational.
• Shortened the max wait time from 6 to 3 weeks

An advantage of this approach is that all users have a more predictable email path. No matter what, we now know all users will get a total of 6 emails over the course of between 5 and 9 weeks.

[joethreepwood]

I'm considering this ready to launch now. I'll give everyone 24 hours to either have a look at the content/flow that they may not have already seen (or to ask for more time), then I'll start rolling it out.

[raquelmsmith]

Cool thanks Joe!