search

PostHog / meta

This is a place to discuss non-product issues in public.
MIT License
10 stars 4 forks source link

Messaging: Experiment engagement test #178

Open joethreepwood opened 4 months ago

joethreepwood commented 4 months ago

Messaging

Context

Raquel suggest the idea of trying a re-engagement campaign for users who repeatedly visit a section of the product without using it.

This tests the tactic by running a holdout test for experimentation.

If a user visits the /experiments section 3 times in 30 days and has a valid email address and has not completed experiment created they are entered into an A/B holdout test.

50% of users get no email (holdout). 50% of users get this email (variation).

Screenshot 2024-02-12 at 14 14 25

The idea is to see if users will be encouraged to start an experiment, measured with the campaign goal of completing experiment created within one week of opening an email in the campaign.

The holdout variant will give us a baseline to judge against. The failure criteria will be a high unsubscribe rate.

raquelmsmith commented 4 months ago

Cool excited to hear the results of this!

joethreepwood commented 4 months ago

Ugh, I actually had this configured wrong. I've set it up correctly now and will check back in a week or two.

joethreepwood commented 3 months ago

I’m going to leave this running for another two weeks, but so far the holdout group who get nothing performs better. 8.6% conversion vs 9.2% conversion. If it continues to bear out this way we may want to think about testing this in a new way that’s more immediate? We could maybe use surveys to give users a prompt.