Start running fortnightly "UserTesting" sessions

[daniellekirkwood]

Collecting feedback via UserTesting in Q1 on mobile was really valuable. We had the ability to:

• measure our SUS Score,
• we found WTFs (impactful small bugs),
• and gather & collate insights all in the bounds of a new user's D1.

As our one of our Q2 projects is to improve D1, we should continue to run the mobile tests, as well as introduce the same test on the web platform.

Still TBD: The platform or method we use to recruit participants and collect data, waiting on Design

Testing questions to ask:

• Your life-long friend Jane Burns has suggested you use a messaging app called Element.
• Download Element to your device using the app store. Please take a look at the imagery and copy on the app store. What is your first impression? Does this look like an app that would be useful to you? Please say out loud why/why not? Once you have downloaded the app, please continue to the next task.
• Once you've downloaded Element, create your account. As you go through the steps, please say out loud if you're finding anything unexpected, difficult or challenging in any way? Please compare this out loud to what you might expect coming from the apps that you use for staying in touch with people for personal and professional communication. After you've created your account, complete this task.
• Now you've got Element, let's find Jane Burns (@janeburns:matrix.org) to let them know you're here. Try to find them and as you do, please say out loud what you expect to happen, what you're finding challenging or easy. Please refer to apps you use when explaining your expectations. Once you've found Jane, and started a chat with them, complete this task. How does that experience compare to other personal messaging apps that you might have used in the past?
• Great! You sent your first message. Now let's explore rooms. For your next task, find a couple rooms of your interest from a list of public rooms available to you. If you are having a hard time finding one, you may also join UserTesting at Element (#UserTestingatElement:matrix.org ). As before, please say out loud what you expect to happen, what you're finding challenging or easy.
• Scroll up and down in one of those rooms, what do you think this room is about? Can you describe what you see on this screen? Try doing the following: 1) Send a message in this room. 2) React to a message sent by someone else 3) Reply to message sent by someone else. How is the experience of sending and interacting with messages? How does that experience compare to other group messaging apps that you might have used in the past?
• Take as much time as you need to explore this app. Move on to the next task when you're ready. What do you think about the different options provided at the bottom? Does something about the app excite you? Does something about the app surprise you? Please share your thoughts out loud.
• Now let's reflect on your experience on Element so far, please rate the following statements on a scale of 1 to 5, with 1 = Strongly Disagree and 5 = Strongly Agree
  • I think that I would like to use this app frequently. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I found the app unnecessarily complex. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I thought the app was easy to use. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I think that I would need the support of a technical person to be able to use this app. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I found the various functions in this app were well integrated. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I thought there was too much inconsistency in this app. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I would imagine that most people would learn to use this app very quickly. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I found the app very cumbersome to use. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I felt very confident using the app. [5-point Rating scale: Strongly disagree to Strongly agree]
  • I needed to learn a lot of things before I could get going with this app. [5-point Rating scale: Strongly disagree to Strongly agree]

Further questions:

• What did you like about the experience?
• What frustrated you most about this experience?
• If you had a magic wand, how would you improve this site?
• How likely are you to recommend this app to a friend or colleague (0=Not at all likely, and 10=Very likely)?

[amshakal]

Consider making the following changes to the flow for web:

Testing questions to ask:

• Your life-long friend Jane Burns has suggested you use a messaging platform called Element.
• Please go to https://element.io/ and launch element web to get started. Please take a look at the imagery and copy on the website. What is your first impression? Does this look like a tool that would be useful to you? Please say out loud why/why not? Once you have launched element web, please continue to the next task.
• On app.element.io, create your account. As you go through the steps, please say out loud if you're finding anything unexpected, difficult or challenging in any way? Please compare this out loud to what you might expect coming from the apps that you use for staying in touch with people for personal and professional communication. After you've created your account, complete this task.
• Now you're in Element, let's find Jane Burns (@janeburns:matrix.org) to let them know you're here. Try to find them and as you do, please say out loud what you expect to happen, what you're finding challenging or easy. Please refer to apps/tools you use when explaining your expectations. Once you've found Jane, and started a chat with them, complete this task. How does that experience compare to other personal messaging apps that you might have used in the past?
• Great! You sent your first message. For your next task, please update your profile picture and display name so your friends and find and recognise you on element. As before, please say out loud what you expect to happen, what you're finding challenging or easy.
• Next up, join a room called UserTesting at Element (#UserTestingatElement:matrix.org) and try doing the following: 1) Send a message in this room. 2) React to a message sent by someone else 3) Reply to message sent by someone else. How is the experience of sending and interacting with messages?
• You have decided to throw a dinner party for two of your friends Jane(@janeburns:matrix.org) and Robert @MXID). Create a room to invite both of them to invite them using element. How does that experience compare to other group messaging apps that you might have used in the past?
• Now let's reflect on your experience on Element so far, please rate the following statements on a scale of 1 to 5, with 1 = Strongly Disagree and 5 = Strongly Agree
• I think that I would like to use element frequently. [5-point Rating scale: Strongly disagree to Strongly agree]
• I found element unnecessarily complex. [5-point Rating scale: Strongly disagree to Strongly agree]
• I thought element was easy to use. [5-point Rating scale: Strongly disagree to Strongly agree]
• I think that I would need the support of a technical person to be able to use element. [5-point Rating scale: Strongly disagree to Strongly agree]
• I found the various functions in element were well integrated. [5-point Rating scale: Strongly disagree to Strongly agree]
• I thought there was too much inconsistency in element. [5-point Rating scale: Strongly disagree to Strongly agree]
• I would imagine that most people would learn to use element very quickly. [5-point Rating scale: Strongly disagree to Strongly agree]
• I found element very cumbersome to use. [5-point Rating scale: Strongly disagree to Strongly agree]
• I felt very confident using element. [5-point Rating scale: Strongly disagree to Strongly agree]
• I needed to learn a lot of things before I could get going with element. [5-point Rating scale: Strongly disagree to Strongly agree]

Further questions:

• What did you like about the experience?
• What frustrated you most about this experience?
• If you had a magic wand, how would you improve this site?
• How likely are you to recommend element to a friend or colleague (0=Not at all likely, and 10=Very likely)?

[amshakal]

With the proposed test above we would be testing the following flows:

1. Finding element web and launching it
2. Creating an account
3. Searching for a matrix ID + sending their first message
4. Finding settings to change picture/display name
5. Finding a room + interacting with timeline
6. Creating a room

[niquewoodhouse]

User tools that have been considered:

• [ ] Usertesting.com (too expensive, hit account limit)
• [ ] Maze (subscription per month)
• [ ] Userzoom seems too expensive

[amshakal]

Tools to consider:

• [ ] https://www.loop11.com/pricing/
• [ ] https://www.userlytics.com/prices-plans
• [ ] https://www.uxtweak.com/recruiting-usability-test-participants
• [ ] https://www.playbookux.com/pricing/
• [ ] https://www.lookback.com/

[niquewoodhouse]

Tools that we can consider:

• [ ] https://www.loop11.com/pricing/
• [ ] https://www.userlytics.com/prices-plans
• [ ] https://www.uxtweak.com/recruiting-usability-test-participants
• [ ] https://www.playbookux.com/pricing/
• [ ] https://www.lookback.com/

I've used lookback a few years back - was really easy to use, will make a trial and try it out on Monday

[niquewoodhouse]

@daniellekirkwood how many sessions did you run on mobile per fortnight please? just trying to work out costs of different platforms, thanks

[daniellekirkwood]

@daniellekirkwood how many sessions did you run on mobile per fortnight please? just trying to work out costs of different platforms, thanks

We ran 4 on each platform every fortnight.

So ideally, in a 2 week period we'd run a test with 12 people; 4 iOS, 4 Android, 4 Web (as a min.)

[niquewoodhouse]

Progress:

• [x] Created an unmoderated test on Lookback for FTUE Web
• [x] Started a feature and pricing matrix to help identify the right user testing tool for us long term
• [x] Started a research brief based on templates for the project
• [ ] Hire users to complete test from various platforms to test
  • [x] User interviews
  • [x] Userfeel
  • [ ] Loop11

[germain-gg]

@amshakal & @niquewoodhouse not quite sure what the next steps are in regards to this task. It does not look like we're taking the direction of running fortnightly sessions. Is this task still relevant and actively worked on?

[daniellekirkwood]

We did not find a way to unblock this so will find an alternative way to deliver the outcome we need.