A/B test: Visitors onboarding tour with search example notebook links

[erzhtor]

Motivation

During discussion on semgrep playground and their playground editor with pre-existing examples, there came a hypothesis to make similar onboarding for newcomers:


2. I like their editor with all pre-existing examples. I think this can be helpful in our case. Our “quick tour” shows a few examples of queries, but we could make a more advanced one with multiple syntax examples. The easiest solution, I could think of is to make an intro/example search notebook with search + description.

NOTE: There seem to be a dedicated web portal for that https://learn.sourcegraph.com. Clarify who owns, it and what purpose it is used for.

Metric and experimental design

Metric: # of activated (or signed up) users, + retention ? Smaller significant change: 10% ? Significance threshold: 5% Duration/size: ?

Descriptive analytics

Other things we will be interested in:

• Individual task completion
• % of visitors who engage with the checklist?
• % of visitors who complete the entire checklist flow?
• % of visitors who complete closes checklist

Test

Experiment/test group: visitors who will get a new tour content with notebook links https://github.com/sourcegraph/sourcegraph/issues/36779, https://github.com/sourcegraph/sourcegraph/pull/36490 Control group: visitors who will get old existing tour content

Flag key: ab-visitor-tour-with-notebooks

[sourcegraph-bot-2]

Heads up @muratsu @a-bergevin - the "team/growth" label was applied to this issue.

[a-bergevin]

@erzhtor @thenamankumar there are a few notebook examples that we could maybe leverage without starting from scratch. Or we can build our own. But I like the idea of reusing. Also these first three notebooks link to eachother at the bottom, which is handy for keeping the learning flow going.

Find Code Across All Your Repos Search & Review Commits Refine queries by filtering Structural search basics

We also need to decide where we'd want to place this. I was thinking maybe we can replace Code Search Use Cases with links to these notebooks on the unauthenticated tour and call it Code Search Basics or Learn to Code Search?

We'd need to skip that "pick a language" step, because it's not relevant here. Maybe instead of having pick a language be a step in the flow we could change it to a dropdown at the top (outside of the steps) that defaults to like javascript or python or something? If users care to change it, great, but if they don't it removes one more step of friction in the flow of the other items. Thoughts?

One unrelated item if we're going to launch an experiment here and open up this part of the code base - how hard would it be to make it so users can "re-open" a completed item in the checklist without getting all the way to the end and restarting? I hypothesize these might be great reference files that people want to revisit before they get through all steps. We could still have the checkmarks to show which things they've visited, but allow them to click and restart any item at any time.

[a-bergevin]

@erzhtor I talked to Emily Chapman and it seems that learn.sourcegraph.com doesn't really have an owner right now. Her team may end up influencing what is on there (and possibly be the owner) but I think we can have a point of view here.

I wonder if we could test linking to some search basics notebooks above, but instead of the link taking you right to the notebook page in full, what if we hosted the notebook on learn.sourcegraph.com and you have a side by side view of the embedded notebook next to search query box / result page.

The notebooks would be read only so you can see the example and run them, but then you can also copy & paste and modify them, run similar queries against other code, etc. Thoughts?

[erzhtor]

Thanks, @a-bergevin for the input! 👍

We'd need to skip that "pick a language" step, because it's not relevant here. Maybe instead of having pick a language be a step in the flow we could change it to a dropdown at the top (outside of the steps) that defaults to like javascript or python or something? If users care to change it, great, but if they don't it removes one more step of friction in the flow of the other items. Thoughts?

1. Yep, the tour doesn't require language if the task/step is not language-specific (which is customizable). So we should be good at this.

One unrelated item if we're going to launch an experiment here and open up this part of the code base - how hard would it be to make it so users can "re-open" a completed item in the checklist without getting all the way to the end and restarting? I hypothesize these might be great reference files that people want to revisit before they get through all steps. We could still have the checkmarks to show which things they've visited, but allow them to click and restart any item at any time.

2. The implementation would be relatively easy/quick, however, it will affect the existing authenticated tour version as well, which is under experiment still. Also, since we don't have designs there are few options on how this could look/work:
   • Making completed tasks unfold with all completed steps, which are also clickable
   • Or putting a "Restart tour", which is available at all times, which will restart the whole tour completed items
   • others...

@erzhtor @thenamankumar there are a few notebook examples that we could maybe leverage without starting from scratch. Or we can build our own. But I like the idea of reusing. Also these first three notebooks link to eachother at the bottom, which is handy for keeping the learning flow going. Find Code Across All Your Repos Search & Review Commits Refine queries by filtering Structural search basics We also need to decide where we'd want to place this. I was thinking maybe we can replace Code Search Use Cases with links to these notebooks on the unauthenticated tour and call it Code Search Basics or Learn to Code Search?

3. I've implemented locally a new visitors tour version with different tasks. Specifically, replaced "Code Search use cases" with two additional tasks https://www.loom.com/share/6d9741f6478942aa8a98319539a0d089:
   • "Code Search Basics" which contains links to the first three notebooks
   • "Advanced Code Search" which contains a link to the 4th notebook I've split it into two groups/tasks because when placing all four links into one, it takes the whole onboarding tour height on the home page, which hides other tasks.

I would appreciate your feedback on the next steps on this?

[erzhtor]

@erzhtor I talked to Emily Chapman and it seems that learn.sourcegraph.com doesn't really have an owner right now. Her team may end up influencing what is on there (and possibly be the owner) but I think we can have a point of view here.

I wonder if we could test linking to some search basics notebooks above, but instead of the link taking you right to the notebook page in full, what if we hosted the notebook on learn.sourcegraph.com and you have a side by side view of the embedded notebook next to search query box / result page.

The notebooks would be read only so you can see the example and run them, but then you can also copy & paste and modify them, run similar queries against other code, etc. Thoughts?

This sounds good. However, it will be more time-consuming I believe. Also, learn.sourcegraph.com is a separate/independent website, so we cannot reuse the existing notebook codebase. We can either:

1. Reimplement from scratch all on learn.sourcegraph.com
2. or create a page/path on the Sourcegraph app (like sourcegraph.com/learn). P.S: However, if we still want to be a separate learn.sourcegraph.com domain, we can configure it to point to sourcegraph.com/learn

If you think we should continue with this a few questions:

1. How do we redirect visitors to this new playground? Possible options: 1) placing "learn" navbar item or 2) putting into quick start tour or 3) CTA/link somewhere else 2) Should we create content/examples ourselves or are there any teams (maybe customer engineering/education teams) who could help with creating examples?