Design a survey for PMs and Leads for CTJ metrics

[stephaniestahlberg]

Dependency

• [x] #179

Overview

As a researcher, we need to create a survey for volunteers who have experience with posting on the Github open roles board (who will mostly but not only be PMs and Leads) so they can evaluate the process for posting roles and recruiting at Hack for LA. These survey responses will generate scores on the CMS metrics we have defined.

Action Items

• [x] Use CMS metrics to design survey questions
• [x] Define the target respondents for the survey
• [x] Make sure that survey questions would serve as appropriate metrics for both the old and new systems
• [x] Discuss the draft survey with the team and HfLA org
• [x] Edit the survey accordingly
• [x] Test the survey with 2 participants
• [x] Make any final edits needed
• [x] Send the survey to target respondents

Resources/Instructions

Research Plan Survey for volunteers who have posted on the Github open roles board

[stephaniestahlberg]

The survey will be sent out at least twice: once to evaluate the old system we currently have in place, and once in the future to evaluate the new system with CTJ once users have had a chance to experience the live MVP and can evaluate it accordingly. The same survey questions will be used so we can compare responses and ratings for the two systems. This Github issue covers only the first time the survey will be sent out.

[stephaniestahlberg]

@sdimran @sharadgaurav7 The dependency on this issue has been closed and I would like to start working on this issue now. Can you move it out of the ice box? Thanks.

[stephaniestahlberg]

@sharadgaurav7 @sdimran @MariaElizabeth04 Please see the survey draft here. After some more thinking, I decided to also include an industry-standard metric for usability that is called the System Usability Score (SUS). The score is based on a 10-item questionnaire, which I included under question 4 in the survey. I have seen that the word product sometimes is replaced by a more appropriate word for the test at hand, such as website, and I replaced it with "process" for our purposes. Besides comparing the current recruiting process to CTJ in the future, another great advantage of using the SUS is that it can be used periodically to evaluate how CTJ is doing and how we measure up against other websites (the SUS is widely used so you can find scores and averages for websites and products online). Please let me know what you think of the survey.

[sdimran]

Hi @stephaniestahlberg, I went through the survey with Bonnie and she is all for running it and getting feedback. she has a few comments and notes that we discussed I am sharing below. @ExperimentsInHonesty if I have missed anything please feel free to add/correct

6 Q3, are we asking based on feedback from people who have joined the team through other means of recruitment?

Q4 is a lot of words with a lot of different asks, might be difficult for the surveyee to keep them all in mind while answering. Would it make sense to break into GitHub, slack and CoP categories to be easier on the eyes. if possible we should include a open text option at the end of every question for users to add additional comments are we predisposing users to have a certain bias to the questions in Q4 by saying ex "I found xyz to be difficult" and then asking to rate agree or disagree. would it make more sense to rather ask "rate the difficulty (or complexity) of xxx from 1-5"

[stephaniestahlberg]

Thanks @sdimran and @ExperimentsInHonesty. I have made changes based on your feedback and also tested our survey with 2 people and made adjustments based on their feedback. From my end, this is ready to go now. Please review and I will send it out when approved.

In response to your comments:

• I added an optional question at the end asking if the respondent would like to share anything else.
• Q3: I have clarified item 6 (which is now item 4) to say "I think that the Github open roles board is hard for Hack for LA volunteers to find". This is how I heard it phrased most often, because people report this challenge is true for new and current HfLA volunteers. People report the only way to find it is to bookmark the page.
• Q4: I simplified the language in the instructions. For the question items, unfortunately, I cannot change the wording because these items come from the standardized System Usability Scale (SUS). I do not think we are predisposing the users to answer a certain way, as this scale has been used for a few decades and has been validated to measure the usability of a system/product. It is considered to be the industry standard, and it is important to keep each item the same to make sure the scale is still valid and reliable.

[stephaniestahlberg]

@sdimran @sharadgaurav7 Can you please review this with Bonnie at your next meeting? My responses to her feedback are listed in the previous comment. Thanks!

[sdimran]

Hi @stephaniestahlberg, this coming Sunday's meeting with Bonnie has been cancelled (04/24) from Bonnie's end. I had to cancel last weeks myself as I was at the emergency room most of the day unfortunately

@ExperimentsInHonesty would you be able to review the above comment from Stephanie and advise if you are okay with the changes from our discussion on 04/03

[stephaniestahlberg]

We reviewed the latest version with Bonnie today and this is good to go. I will send it out to target respondents tomorrow, Monday, and then I will close the issue. If we need help getting more respondents, Bonnie offered to send it directly to people (I think PMs) who were previously active but right now are not in a team.