Email: Session recording upsell experiment

[joethreepwood]

We haven't yet tested our ability to use email to upsell users into other features/products within the platform.

I'm biasing for impact and advocating that we should test it. Email is a potentially powerful and encouraging users to engage with other features/products they haven't used yet is only going to become more important give our product vision.

That said, it's a bit salesy, so feedback welcome.

How

Recordings are currently less popular than insights. So, we should be able to unlock the biggest impact by encouraging users to find value in session recordings.

I'm planning to send this on Friday, with a delay set for Monday 6th March delivery.

We'll target users according to the following criteria and direct them to app.posthog.com/recordings:

Users who fit all of the below criteria:

• Performed User Signed Up
• Have not performed Recording Analyzed more than 3 times since signing up
• Have a valid email address
• Are on US Cloud
• Are recently engaged (have opened any email from us in last 180 days)
• Is subscribed to the Actually useful marketing emails category

I've added the US Cloud + Recently engaged email to help simplify the CTA and target users who respond well to emails from us. It's a segment of around 3,500 users total.

What

I'm proposing three variants for the email. As a conversion goal we'll target getting users to perform recording analyzed within 10 days of receiving the email.

We can't target/track users based on billing for this at the moment, because that's an event which occurs at the org level, rather than for all users in an org. We'll fix that in time; it shouldn't block us now as recording analysed can occur at the individual level.

Content will be finessed a little further, but is ready for feedback.

Variant 1: Features > Benefits

Variant 2: Illustrating a use-case

Variant 3: Benefits > Features

Open to feedback on all aspects here

[andyvan-ph]

Maybe Try session recording or Enable session recording for the CTA? Where is the CTA going?

[raquelmsmith]

Are on US Cloud

Why not EU? I imagine it should be easy enough to change the CTA to direct to EU cloud if needed.

Variant 2: Illustrating a use-case

I feel like the first tip (inviting team members) feels really unrelated. Would change the tips and order to something like:

1. Enable recordings on your marketing site
2. Create a session recording schedule
3. Share recordings with other team members by inviting them for free

Variant 3: Benefits > Features

Maybe tweak: "Understand why, where, and how users get stuck"

Each sentence ends with an exclamation and it reads kinda weird.

I like how this one includes info about free allotment - I'd add that to all of them.

--

Did the zapier idea not float your boat?

[joethreepwood]

Maybe Try session recording or Enable session recording for the CTA? Where is the CTA going?

Recording is pointing directly to app.posthog.com/recordings

Why not EU? I imagine it should be easy enough to change the CTA to direct to EU cloud if needed.

We can do this, for sure. Simplest way is duplicating the broadcast for an EU audience and sending to eu.posthog.com/recordings. But that was an extra step of work which didn't feel needed for a test like this.

Happy to test on EU too, if you feel strongly it's worth seeing how a EU audience responds. My plan was more: test on US, find a winner, test that on EU.

Variant 2: Illustrating a use-cas Variant 3: Benefits > Features

👍

Did the zapier idea not float your boat?

Potentially it did, but it feels like it'd be a bit of work to investigate - and from speaking to @kappa90 he may be about to prioritise updates to the Customer.io integration so I've parked it for now. It doesn't feel like it would block this experiment unless you disagree with the metric?

[joethreepwood]

I'm planning to send this on Friday, with a delay set for Monday delivery FYI.

[joethreepwood]

[raquelmsmith]

unless you disagree with the metric

No, I think getting people to simply use session recordings is enough.

[jamesefhawkins]

Email content

Subjective feedback, but I prefer the first email by a way here - it's more on brand.

The case study feels super marketing-y - I think these are valid on the website as an initial impression of us (ie these signify "posthog has customers", and if someone reads them, they're pretty good), but in an email I'd instantly spam this. I think I'd probably also do this for the benefits oriented one. However, the first one gives a valid reason to try session recording.

You could even split test your title suggestion versus ie: "You can now track page performance with PostHog's session recordings", "You can now track console errors with PostHog's session recordings", then to explain how to do these things, plus the extra features available. It'd give more incentive to use session recordings + would raise awareness. The third new feature - sharing - isn't really a relevant feature for someone not using session recording since why share something even you don't care about.

I think you could put some benefits in, perhaps at the end of the email ie 'Why use session recordings?' then a few bullet points. In general, I think features first, benefits second is the way to market to our audience.

Small nit:

This isn't accurate for the segment you're targeting - they may have used it a few times.

Emails in general

I've seen zero evidence that these affect user behavior for us, but they definitely could be annoying our user base. I'd be way keener on emailing like this if we've a little data to back up that there is benefit to them, else I'd rather not risk lowering our open rates. This email feels like an easy experiment to test, but I thought we were already going to do this for the activation chain of messages we send. I'm a bit out of the loop here, no need to explain it to me - exercise your own judgement.

[joethreepwood]

Thanks James - I'll cut the case study variant for now and test the other two against each other and a holdout version. That'll give us the best read on data. We'll find out

Emails in general

I've seen zero evidence that these affect user behavior for us, but they definitely could be annoying our user base. I'd be way keener on emailing like this if we've a little data to back up that there is benefit to them.

Just want to address this specifically - I don't feel we've tested this for email yet, so this is really the opportunity to get that data.

In terms of impacting behaviour, I looked at this here for the onboarding campaign. Short version: The onboarding email flow has a 14% conversion rate for helping users who we think were otherwise blocked to ingest their first event. I'm currently refocusing email onto other, higher-impact metrics, but can chat further if helpful.

[joethreepwood]

There's still some time left on the goal window (we set it for 10 days from an open, we're only 4 days in since send), but to give a quick snapshot of the results:

The variants are currently basically equal. There's a 1.9% conversion rate, which is around average. The benefits > features version is actually slightly more popular, but only by one conversion.

My read on this if these results stay consistent would be that we can impact some user behaviour with emails to within the normal limits -- but we will want to be quite judicious with how we deploy it given the amount of noise created.

A good follow-up experiment here may be to further filter down into subgroups. Do High ICP groups respond differently?

[joethreepwood]

Final follow-up on this - the conversion rate has risen to 3.1%.