New user's journey [Web]: Delight

[daniellekirkwood]

This issue is WIP while the team defines their plan ahead.

Intro

Problem we’re solving

• Currently we offer no guidance as to the steps new users should take in product in order to get value in their first session. We also learned in our second round of testing that the learning curve for Element is steep.
• Using a list of hypotheses and rapid experiments intended to simplify the journey for new users, we will improve D1. While conducting these experiments we would collect data that helps us to understand better the new user journey and flows that need to be revised later.

Goals

• Understand the web user's journey when they're new to Element
• Improve the user's experience in their first session

Non goals - Problems we’re not solving

• Changing fundamental flows; We want to first understand and learn the impact of actions and communication to D1, then we can move to improving the flows if needed.

Guiding principles

• Move fast; Be able to undo quickly anything that isn't working
• Be clear on the goals before starting; Our ability to measure experiments is limited, by being very clear on the things we want to see we are able to call pass/fail with more confidence.

Success Metrics Quantitative:

• Improve Web D1 retention
• Do not decrease D7 or R30 retention
• Increase "Start tasks" and "Task completion" rates in first session

Qualitative: TBD

---

Discovery

Discovery output

• Which tasks are conducive to higher retention rates, and which aren’t?
• Which tasks do users struggle with the most and what are the suggestions for improvement?
• How do we encourage users to engage with high-retention flows?
• How might we tailor a user's first session to ensure the highest chance of success?

Delivery

• Run UserTesting sessions over time to establish a baseline and measure progress
• Craft a list of hypothesis and assumptions that we think will improve D1
• Deliver rapid experiments that improve D1 on Web

Risks to mitigate	Risk	Details	Mitigation options	Decision
Increasing "Start task" decreases D1	We may find that pushing users into task completion reveals issues with tasks that need fixing before we see significant improvement to retention or first session success.	These would require prioritisation review discussions at milestone steps in the future.
Running sound experiments is not easy	- We have no A/B test infrastructure

• The user base is potentially hostile towards the idea of changing the software based on calling home
• We lack the user numbers to quickly reach statistical significance for small changes in retention
• We don't have anyone with a stats background to ensure experiments are well designed | Don't run A/B tests, instead be clear about impact and hold ourselves accountable to that. | |

---

The Experimentation Plan

TBD: The team needs to meet and list the assumptions and hypothesis we have

[daniellekirkwood]

Decision made: We will prioritise this project, starting Q2

The web team is driving towards the D1 metric, aiming to make the biggest impact we can through learning and experimentation over time.

Next steps: April 12, 2022

• [x] Kick off with the team that: defines how we work together, the assumptions and hypothesis we have so far, project milestones
• [ ] Run user research sessions on web for customer empathy and insights in current project, this also establishes a baseline
• [ ] Decide and confirm how we will approach our experiments and analytics in order to measure what we ship
• [ ] Build and ship!

[daniellekirkwood]

Project Kickoff April 20, 2022

New User's Journey next steps

• [ ] Ensure Meta issue up-to-date
• [ ] Organise the learning plan with the questions from the team
• [ ] Synthesise the session from project kick-off
• [ ] Define Experiment template and the go/no-go decision points for each test
• [ ] Work in PostHog

Inputs to the project

• Internal team members ideas and suggestions
• Analytics and PostHog data (https://github.com/vector-im/element-meta/issues/193)
• Track the current home button usage (https://github.com/vector-im/element-web/issues/21705)
• UserTesting and UserZoom feedback

[daniellekirkwood]

Week ending April 29, 2022

• Experiment plan
  • Please add your votes using your initials, capture your ideas by adding a new row to the table, and your thoughts in comments
• While we're working on the designs for bigger swing experiments, here's a list of WTFs we can fix now
  • Please use the Google sheet as a reference, but keep comments in the GH issues directly.
  • If you have an idea for a WTF, please add it to the list
• The learning plan draft shows our list of questions and how we might answer them
  • Please leave comments
• This is the figma we've been using to discuss our thoughts while grounded in the user's experience

[daniellekirkwood]

This remainder of this project has been deprioritised in place of Threads. We delivered the New User Checklist: https://github.com/vector-im/element-web/issues/22463

Moving this back to Concept.