search

PostHog / meta

This is a place to discuss non-product issues in public.
MIT License
18 stars 4 forks source link

Messaging: Replay Recommender #253

Closed joethreepwood closed 3 weeks ago

joethreepwood commented 3 weeks ago

Messaging

The trouble with replays is that it's easy to forget to use them and it's hard to find the signal amidst the noise.

We've got some solutions to the latter now (activity scores), so here's a solution to the former: An email campaign that tells people who forget that they've got some cool new replays to watch.

Workflow

Users will enter this campaign if:

This ensures we are reaching people who have recordings to watch, but aren't watching them.

Once they're in the flow, we immediately trigger the following email:

Screenshot 2024-10-29 at 17 01 33

If they click a link in that email, we then set replay_recommender_conversion to true on the person The link we drive them towards is a list of activity score recordings in the last 24 hours which are greater than 30 seconds.

They then exit the campaign. They can't enter it again for another 4 weeks. This stops the emails becoming annoying.

Subscription changes

We want to give users a way to opt-out of these emails without opting out of other emails. So, I've created a new subscription type and updated the existing ones.

Screenshot 2024-10-29 at 15 31 41
joethreepwood commented 3 weeks ago

@raquelmsmith open to your thoughts here on what's worth adding an experiment for, etc

raquelmsmith commented 3 weeks ago

Nice! Love the hogbot 😍

I wonder if it's worth telling them about the "what to watch" page? Eg. "Need some other inspiration? Check out our what to watch page." Or could cycle through specific recommendations - "want another idea? how about watching users who used your signup flow?" and then we could highlight the specific template when they reach the page. (I'm okay with it if you think it's a bad idea, but since the goals are similar - help people see the variety of ways they can use replays - it might go well together)

An experiment would be good.

Control group does not get the email, prop on person (or event? I'd kind of rather do this with an event) that says they were entered into the experiment with a variant control.

Test group does get the email, prop/event saying they were entered into exp with variant test.

I'd make the control group 20% of the users and the test 80% (since most prob won't click anyway - though receiving the email at all could remind them that it exists!)

joethreepwood commented 3 weeks ago

Nice! Love the hogbot 😍

I wonder if it's worth telling them about the "what to watch" page? Eg. "Need some other inspiration? Check out our what to watch page." Or could cycle through specific recommendations - "want another idea? how about watching users who used your signup flow?"

This is there are the moment, albeit buried in the footer. We can move it out, but I think we still want it minimized -- having two CTAs which are boldly put together is going to complicate the messaging, I think.

Screenshot 2024-10-30 at 10 43 42

An experiment would be good.

Control group does not get the email, prop on person (or event? I'd kind of rather do this with an event)

Do we need an event to do this? I can easily configure the experiment in Customer.io and track success there, without it. I'll set it up now, but the Customer.io integration isn't two-way -- so I don't think this event will make it back to PostHog anyway.

joethreepwood commented 3 weeks ago

All set up. We'll set an event and see what happens with that, but the A/B test in Customer.io should give us a decent read anyway and I can loop back with the results once it's running.

Screenshot 2024-10-30 at 10 54 54

Unless any concerns, I'll set this to run tomorrow morning? We'll enter the 5,000 current matches immediately.

raquelmsmith commented 3 weeks ago

🙌 awesome! What's the event that's being sent?

joethreepwood commented 3 weeks ago

🙌 awesome! What's the event that's being sent?

replay_recommender_conversion

Screenshot 2024-10-30 at 15 59 55
raquelmsmith commented 3 weeks ago

Oh is this sending an event/updating the person in customer.io? I meant an event/prop in posthog 😄

joethreepwood commented 3 weeks ago

Yeah, it's a C.io event -- I don't think we currently have the ability to send stuff to PostHog, but I am working on it. I'm speaking with Customer.io and Olly from CDP team soon about getting PostHog integration setup for their data pipeline so we can push data back to PostHog.

For now, we can monitor the experiment results and initial impact in Customer.io - and if we want to do a deeper dive into behaviours in a few weeks then let me know and I'll manually bring the data back to PostHog as an export so we can run it there still.

Launching this today!

joethreepwood commented 3 weeks ago

We hit a snag with this.

One user replied saying that he didn’t use replays at all. Logged in as him and confirmed that he had no replays at all. Debugged with Annika and we can’t see any real usage from him in the last 12 weeks, no recordings on file.

We've realized this is due to the organization_usage_report segment not having a filter on it. I've now added a filter saying that that event must have come in the last 21 days, which will hone it in.

All these emails are coming directly from me at the moment, so I'll respond to any users who flag this.

joethreepwood commented 2 days ago

30 days later, tracking a 7% conversion