A/B test: Authorized Onboarding checklist effectiveness

[ryankscott]

Motivation

We are looking to move the Getting Started Checklist for visitors to those users who've signed in. This work is designed to help users understand Sourcegraph and hit their "Aha" moment, resulting in longer-term continued engagement with the product.

Test

Experiment/test group: users who will get a new tour for authenticated users https://github.com/sourcegraph/sourcegraph/issues/28322 Control group: users who will get regular untouched UI without new tour for authenticated users

Metric and experimental design

Metric: # of searches performed by a user over 30 days Smaller significant change: 10% (?) Significance threshold: 5% Duration/size: We need roughly 1.1k per test. Current new weekly signup rate => ~8 weeks

Descriptive analytics

Other things we will be interested in:

• Individual task completion
• % of users who engage with the checklist?
• % of users who complete the entire checklist flow?

Flag

key: quick-start-tour-for-authenticated-users

[erzhtor]

Created a 50% rollout feature flag in the admin panel. cc @ryankscott

[sourcegraph-bot-2]

Heads up @muratsu @a-bergevin - the "team/growth" label was applied to this issue.