User Testing

[SachaG]

Tasks for user testing:

Scenarios

• Platform (pick 1)
  • Mobile
  • Desktop
• Account Type (pick 1)
  • Log in with email & magic link
  • Take survey as guest

Tasks

Complete the following tasks while:

1. Recording your screen
2. Narrating your thought process out loud

• (Optional) Select a different language
• Access survey landing page (https://survey.devographics.com/en-US/survey/state-of-css/2023)
• Create account or continue as guest
• Fill out the survey normally
• Add at least one feature to the Reading List
• Leave at least one comment
• Fill in at least one "other…" answer
• Upload screen recording to Dropbox or similar and leave a link in this thread

!!!IMPORTANT!!!

You should assume this recording will potentially be public. Do not share any information (salary, etc.) that you are not comfortable disclosing publicly (skipping these questions or answering them randomly is fine).

[michaelquiapos]

Hi @SachaG,

Part of the User Research Plan is the Pretest or Pilot testing. The objective there is to understand the user experience of the target audience. The target audience for this survey are CSS users of all levels of experience, including:

• Web developers
• Graphic designers
• Content creators
• Anyone who uses CSS

Remote user pretests are a way to gather feedback on a product or service from users who are not physically present. This can be done through various methods such as online surveys, video calls, or screen sharing sessions.

Some ways for conducting remote user pretests include:

1. Clearly define the goals and objectives of the pretest.
2. Use a well-designed questionnaire or interview guide to gather feedback.
3. Choose the right participants who match the target user group.
4. Use appropriate tools and platforms for remote testing.
5. Ensure that participants have a comfortable and distraction-free environment.
6. Record the sessions for later analysis and review.

I'd like to include an opening screen to set expectations. Something like this: _"Thank you for choosing to take this important survey about the State of CSS. It should take about 20 minutes to complete.

Your participation in this research is voluntary. Your responses may be shared with the research sponsor for purposes of research-related analysis, but only after we have removed any information that identifies you personally. Click here to view our Privacy Policy.

Under no circumstances will information you share with us be used to directly sell or market any products or services to you.

Do not use the Back or Forward arrows on your browser to move through the survey. Please do so using only the arrows at the bottom of your screen.

Click on or tap the "Forward" arrow when you are ready to begin your survey."_

[michaelquiapos]

Here I've made expansions with your Scenarios.

Scenario 1: Mobile Medium, Log in with Email Account Type

1. User receives an email invitation to take the pretest.
2. User clicks on the link provided in the email and is directed to the pretest landing page.
3. (Optional) Select a different language
4. User enters their email address and password to log in.
5. User is directed to the pretest questionnaire.
6. User completes the pretest questionnaire.
7. Add at least one feature to the Reading List.
8. Leave at least one comment.
9. Fill in at least one "other…" answer.
10. User submits their responses.
11. Upload screen recording to Dropbox or similar and leave a link in this thread.

Scenario 2: Mobile Medium, Magic Link Account Type

1. User receives an email invitation to take the pretest.
2. User clicks on the link provided in the email and is directed to the pretest landing page.
3. (Optional) Select a different language
4. User clicks on the magic link button.
5. User receives a text message with a magic link to access the pretest questionnaire.
6. User clicks on the magic link and is directed to the pretest questionnaire.
7. User completes the pretest questionnaire.
8. Add at least one feature to the Reading List.
9. Leave at least one comment.
10. Fill in at least one "other…" answer.
11. User submits their responses.
12. Upload screen recording to Dropbox or similar and leave a link in this thread.

Scenario 3: Mobile Medium, Guest Account Type

1. User receives an email invitation to take the pretest.
2. User clicks on the link provided in the email and is directed to the pretest landing page.
3. (Optional) Select a different language
4. User clicks on the guest button.
5. User is directed to the pretest questionnaire.
6. User completes the pretest questionnaire.
7. Add at least one feature to the Reading List.
8. Leave at least one comment.
9. Fill in at least one "other…" answer.
10. User submits their responses.
11. Upload screen recording to Dropbox or similar and leave a link in this thread.

Scenario 4: Desktop Medium, Log in with Email Account Type

1. User receives an email invitation to take the pretest.
2. User clicks on the link provided in the email and is directed to the pretest landing page.
3. (Optional) Select a different language.
4. User enters their email address and password to log in.
5. User is directed to the pretest questionnaire.
6. User completes the pretest questionnaire.
7. Add at least one feature to the Reading List.
8. Leave at least one comment.
9. Fill in at least one "other…" answer.
10. User submits their responses.
11. Upload screen recording to Dropbox or similar and leave a link in this thread.

Scenario 5: Desktop Medium, Magic Link Account Type

1. User receives an email invitation to take the pretest.
2. User clicks on the link provided in the email and is directed to the pretest landing page.
3. (Optional) Select a different language.
4. User clicks on the magic link button.
5. User receives a text message with a magic link to access the pretest questionnaire.
6. User clicks on the magic link and is directed to the pretest questionnaire.
7. User completes the pretest questionnaire.
8. Add at least one feature to the Reading List.
9. Leave at least one comment.
10. Fill in at least one "other…" answer.
11. User submits their responses.
12. Upload screen recording to Dropbox or similar and leave a link in this thread.

Scenario 6: Desktop Medium, Guest Account Type

1. User receives an email invitation to take the pretest.
2. User clicks on the link provided in the email and is directed to the pretest landing page.
3. (Optional) Select a different language.
4. User clicks on the guest button.
5. User is directed to the pretest questionnaire.
6. User completes the pretest questionnaire.
7. Add at least one feature to the Reading List.
8. Leave at least one comment.
9. Fill in at least one "other…" answer.
10. User submits their responses.
11. Upload screen recording to Dropbox or similar and leave a link in this thread.

[michaelquiapos]

Finally, the following are some important reminders that user pretest participants should know and how to prepare them based on scenarios 1 to 5:

1. Make sure that participants have a stable internet connection to avoid any interruptions during the pretest.
2. Inform participants about the estimated time it will take to complete the pretest questionnaire.
3. Instruct participants to use a quiet and distraction-free environment to ensure focus and concentration.
4. Remind participants to read and follow the instructions carefully to avoid any errors or confusion.
5. Provide clear guidance on how to navigate the pretest platform, including how to log in and access the questionnaire.
6. Encourage participants to be honest and provide feedback based on their true experiences and opinions.
7. Remind participants that their responses will be kept confidential and used only for research purposes.
8. Provide a way for participants to contact you in case they encounter any issues or have questions during the pretest.

For scenarios 1 to 5, here are some additional reminders:

Scenario 1: Mobile Medium, Log in with Email Account Type

• Remind participants to check their email for the invitation and to use the same email address and password to log in.
• Instruct participants to use a mobile device with a compatible browser to access the pretest platform.
• If the email was not on their mail client inbox, have them check the spam folder.

Scenario 2: Mobile Medium, Magic Link Account Type

• Remind participants to have their mobile phone ready to receive the text message with the magic link.
• Instruct participants to click on the magic link to access the pretest questionnaire.

Scenario 3: Mobile Medium, Guest Account Type

• Inform participants that they do not need to log in and can access the pretest questionnaire as a guest.
• Instruct participants to use a mobile device with a compatible browser to access the pretest platform.

Scenario 4: Desktop Medium, Log in with Email Account Type

• Remind participants to check their email for the invitation and to use the same email address and password to log in.
• Instruct participants to use a desktop or laptop computer with a compatible browser to access the pretest platform.
• If the email was not on their mail client inbox, have them check the spam folder.

Scenario 5: Desktop Medium, Magic Link Account Type

• Remind participants to have their mobile phone ready to receive the text message with the magic link.
• Instruct participants to click on the magic link to access the pretest questionnaire.
• Instruct participants to use a desktop or laptop computer with a compatible browser to access the pretest platform.

[SachaG]

That's great! My only feedback is that it might be better to err on the side of not giving too much instructions, so we can see what problems actually happen in a natural context?

[michaelquiapos]

Remote testing is a way to see how people use the survey. Try to make it like how they would use it in a natural context. Not giving too much help to see what they would do on their own is the way to go. Yet it still is practical to tell them how to use the testing site and what they need to do. Also, explain why we are doing the test so they know what to expect.

When we want to see how people use a website or product, we don't want to tell them what to do too much because then it won't be natural for them. We want to see how they would use it on their own. This helps us find any problems and make it better for everyone. We just need to give them enough help so they understand what to do.

[michaelquiapos]

I can see how you may wanna choose an unmoderated testing in this scenario. It is cheaper and faster. You might expect an unmoderated study to cost anywhere from 20% to 40% less than a moderated equivalent. The researchers might also save around 20 hours of time. But there are tradeoffs associated with using unmoderated testing.

You would wanna watch out for people who just want to get paid and move on or who participate in studies on a regular basis. To be safe, I typically recommend overrecruiting a bit for about 10% to be safe. So instead of testing 5 participants in an unmoderated study, you might wanna run the test to 6 participants.

You might find that whatever time you’ve saved in conducting the session will be shifted into the analysis phase of the project. Remember, particularly in qualitative testing, you should still be watching those recordings to analyze them. Most testing platforms have short limits on how long sessions can run. If you would use a testing platform there are premium costs for longer recordings. For this reason, long, complex tasks may not be well suited to unmoderated testing. As sessions are planned to an estimate of 20 minutes. If you are planning to test many tasks or that the survey length could be double the planned session estimate, you may need to recruit more participants (with each participant doing only a subset of the tasks).

Consider all the specifics of your situation (tools, schedule, users, and recruiting) and weigh those against the research goals when you make your choice.

[sarajw]

Am currently using Arc on MacOS Ventura 13.3.1, started survey as a guest, got a client_data_validation_error error at the end of the first page, stopped the testing early:

https://drive.google.com/file/d/1mXwkV30wfQ90UzMMgWTY5OEs_93-Vl5q/view?usp=drive_link

[SachaG]

Oh that’s strange. Any chance you could check in the dev tools and paste the contents of the ‘saveResponse’ request that’s triggering the error?

[sarajw]

It doesn't seem to be finding saveResponse - please assume I'm dumb and tell me exactly where else to look in devtools if I've missed something!

EDIT: It works now, I just tried to click again, and it went through :) Did you change something?

[SachaG]

It seems like this was due to a mismatch between the survey contents as defined locally in my dev environment and the same info as stored in our production API. So sometimes the survey was trying to submit a question that didn't exist anymore, which threw an error.

I think that specific problem is fixed, but let me know if you run into any other issues. Sorry about that!

[sarajw]

Here's the rest of it - sorry it's over 20 mins long! https://drive.google.com/file/d/1IIlbPspZMMP4ZEWbRyGgHKwfNqujCY4G/view?usp=sharing

Main comments:

• A light mode would be nice
• Some things are hard to answer when you are not in charge of what tech you use at work
• Other things are hard to answer when you're a long-time hobbyist but new to working in the web industry
• I love the reading list functionality

I had no further errors :)

Edit: maybe just one, this 'undefined' in the reading list email, see image:

[SachaG]

@sarajw thanks so much! it's super useful to see somebody else interacting with the survey, it always helps me find things to improve :)

[SachaG]

And yes we need a Mastodon share button!

[sarajw]

You're very welcome :)

[ShaswatPrabhat]

Hi all,

Maybe this has already been discussed but just checking on something below:

For section Headings being auto highlighted based on the section under question there is a good chance that the last section will never get auto highlighted, I have seen the same in a lot of other sites but wanted to call it out, below text-wrap: balance in the section headings will never get auto highlighted :

[ShaswatPrabhat]

On the About You page, I do not see an email validation on entering or blurring the same:

[SachaG]

@ShaswatPrabhat good point, it's true there is no email validation.