Growth: Activation Help Emails - Feedback

[joethreepwood]

@raquelmsmith asked me to in https://github.com/PostHog/meta/issues/172 to have a look at the activation help emails they have set up. I wanted to move it to a new issue to avoid cluttering that one.

Activation help emails - running a couple experiments here, one seems to do something, the others don't. Curious if you have more data.

You've got recordings

Currently looks like this:

This experiment seems to be working well and has good response, but I think we could improve it.

• Small thing: You're missing any pre-header text and including the URL directly isn't the best practice: better to put the link on relevant text or to include a button.
• This email is great and seems to perform OK, but I think it could offer more value. I have a feature request out for some Replay templates. If we had those we could directly link users to playlists that show their recent rageclicks, etc.
• The experiment here is configured incorrectly. It looks like you want to run a hold-out test, but you're running it with a branch within the workflow instead of doing it as an A/B test. Functionally this won't change much, but switching it to a holdout test will get you a clearer read on the data.
• You're sending a personal email style but using the PostHog design template. Best practice is to use one or the other - it's probably worth testing which works best. As it is, it feels like the email is a little spammy.
• The email probably isn't giving users much value because it only triggers based on if they have ingested recordings and nothing else. So, they may not have new recordings or even any recordings they haven't seen. We could solve this by tweaking the trigger: make it so a campaign triggers when a user has ingested events and not logged in in 4 days.
• I'd suggest switching the email to a new design personally, including an image and a bit more detail, e.g:

Product analytics insight activation

This doesn't seem to be performing well.

• It could be better set up as an A/B test, and I think we could change the style to be either a personal or designed email instead of both.
• Again the URL should change to a button. Currently it specifies us.posthog but we aren't just targeting users on that domain. The link will auto-redirect, yes, but users will bristle if we send them unpersonalized content like that.
• 6 hours feels like too short a time to be sending this email to users. These users will already have had the welcome email 5 hours before this. 6 hours later they'll then get the next onboarding email/newsletter invite. I'm not sure this is offering enough value over the welcome email (which already directs to the dashboard template and AARRR guide).
• I think the content should offer much more value to the user. Simply directing them to the insights page isn't very useful and doesn't give them much to go on. I think if we direct users to a dashboard here as a way to quickly start getting value then that'd be better, and we can support insights with a goal. The email should make it very easy to get started with the behaviour we want to encourage.

Here's what it could look like - however I ultimately think we can retire this experiment in favour of the welcome email.

Product Analytics Ingestion Activation

• Many of the same issues above.
• This email is already basically replicated by the welcome email, which sends after 1 hour (not six) and directs users to the exact same pages (except for inviting a teammate). The existing welcome email has an open and CTR rate of 59% and 8% to this emails' 46% and 0.9%.
• Based on this I don't think it's a useful test. Instead, I'll update Onboarding 5.0 to send two variants of the Welcome Email based on role_at_organization and direct users who are not engineers to invite a team mate.
• We've tested versions of this email before in previous email onboarding flows to encourage ingestion and didn't get strong results with email, FYI.

[raquelmsmith]

Thanks for taking a look at this!

Small thing: You're missing any pre-header text and including the URL directly isn't the best practice: better to put the link on relevant text or to include a button.

That was actually intentional as I feel full links are often more click-able, but that was a gut feeling for me and I don't have data on it.

The experiment here is configured incorrectly. It looks like you want to run a hold-out test, but you're running it with a branch within the workflow instead of doing it as an A/B test. Functionally this won't change much, but switching it to a holdout test will get you a clearer read on the data.

I had it set up so we could evaluate results in PostHog: https://us.posthog.com/experiments/9476

If you think there is a better way to do it let me know and/or feel free to just change it.

You're sending a personal email style but using the PostHog design template. Best practice is to use one or the other - it's probably worth testing which works best. As it is, it feels like the email is a little spammy.

I wanted the plain text version but didn't see it as an option so I just did this. Please update it if you'd like!

The email probably isn't giving users much value because it only triggers based on if they have ingested recordings and nothing else. So, they may not have new recordings or even any recordings they haven't seen. We could solve this by tweaking the trigger: make it so a campaign triggers when a user has ingested events and not logged in in 4 days.

We make sure they have at least 10 recordings and have watched fewer than 5:

I think waiting 4 days is too long, but we can run an experiment.

I'd suggest switching the email to a new design personally, including an image and a bit more detail, e.g:

Plain text-emails generally perform better, ya? Have you tried variants of plain-text vs formatted/imagey emails and the formatted ones do better for our audience?

[joethreepwood]

Cool. I've been through and made these fixes, but left them without A/B testing since you've got that managed. In the future you can just do this directly in Customer.io, but if I were to implement that now it would be starting from scratch and invalidating results.

Note that I've also changed the FROM address for each email. Most of these emails referenced that users could hit reply and you'd respond -- but the FROM address meant that these emails would be sent from hey@posthog.com. Now they're sent from your personal emails, so replies actually will come back to you.

I've also added preheader text and the other changes I mentioned.

You can apply the basic email template by applying the empty layout, btw.

We have tested plain vs designed emails before, along with other things. This does perform better for certain types of emails, but it's often a minor difference. Plain emails tend to perform better for invitations and personal offers, but for things like the recording email I think a designed email would perform better -- but up to you!

The emails now mostly look like this - the rest of the feedback is up to you if you want to take, so I'll close this for now!