joethreepwood
commented
7 months ago I've rolled this out for events now. I'm looking at rolling it out for Surveys and Flags too.
@raquelmsmith Can you give me your quick gut feel one what the right value for flag requests may be here?
Surveys I'm going to set up as a very low figure - around 5 responses. Responses feel less common as a count within this timeframe.
Onboarding changes are long, complex, and hard to explain. I'll deliberately keep this as brief as possible.
Context
I’ve been running an experiment in the Onboarding 5.0 flow for about two months. We had an A/B test running that offers two routes for upselling users to replays.
First, we checked if a user had ingested enough recordings. If yes, we gave them an upsell message as normal. If not, we gave them one of two variant messages. The goal was to
billing product activated, but really we were using that as a proxy for how well we can unblock users who aren't getting data in.One variant (A) was focused on selling the features by talking about a case study. The other (B) was focused on helping users setup by giving install instructions.
What we found was that (A) had a better open rate, but (B) had a better conversion rate. The raw numbers were small, but eventually we got enough confidence to push forward. I merged the subject line of (A) into the email for (B) and made some slight tweaks to make it make sense. This gave us the best performing subject line, and the best performing copy, together.
Changes
Onboarding 5.1 changes the flow to adopt a similar decision across other flows, not just replays. We use the copy for replays as a basis. In order to accomodate the expectation that this data exists accurately enough, I've increased the delay before this stage of the flow from 24 hours to 48.
E.g., For analytics, we now check if a user has ingested enough (50) events in the period. If Yes, we give them the current upsell. If not, we give them a combination email based on the replay results.