jessvb
commented
4 years ago User Study Questionnaires Opening Survey 1) How old are you? 2) What is your gender? → male, female other (with text box), prefer not to answer 3) What is your race and ethnicity? → can select multiple, other (with text box), or prefer not to answer 4) Is English your first language? → yes, no, yes along with another language 5) Would you consider yourself a Novice or Advanced programmer? (See definitions below)
- Novice: no programming experience, or have experience in blocks-based programming (e.g., Scratch, App Inventor)
- Advanced: have experience in object-oriented programming or completed a course or project in a text-based language (e.g., Java, Python, C++) 6) What programming languages and/or software have you used before? (Select as many as applies) → none, can select multiple, other (text box) 7) What type of conversational agents have you interacted with before? (Select as many as applies) → none, can select multiple, other (text box) 8) I am a programmer (5-point Likert scale)
Survey After Each Activity (for novice & advanced) Note that [system] will correspond to the particular activity completed; i.e., the “voice-based system”, “text-based system” or “voice- and text-based system”.
Likert scale: strongly disagree, disagree, neutral, agree, strongly agree
- I found it difficult to complete the goal with [system].
- I found programming with [system] difficult to use.
- I am satisfied programming with [the system].
- I found programming with [system] efficient to use. Short answer free responses:
- What did you like about programming with [system]?
- What was frustrating about programming with [system]? How could we make it less frustrating?
- What did you wish you could say to the agent? What didn’t the agent understand?
- What features can we add, change, or remove to make the system better?
Overall Comparative Survey (after all activities have been completed)
- Did you prefer: voice ←------→ voice+text (5-point slider)
- Did you prefer: text ←------→ voice+text (5-point slider)
- Did you prefer: voice ←------→ text (5-point slider)
- Likert scale: strongly disagree, disagree, neutral, agree, strongly agree:
- I think programming with just voice is easier than programming with just text.
- I think programming with just text is easier than programming with just both voice and text.
- I think programming with just voice is easier than programming with both voice and text.
- I think programming with just text is frustrating or hard.
- I think programming with just voice is frustrating or hard.
- I think programming with both voice and text is frustrating or hard.
- I enjoyed the process of trying to complete my goal.
- I think being able to program using voice is useful.
- I like being able to program using voice.
- If I could, I would continue to learn how to program using voice.
- I am a programmer.
- Short answer free responses:
- We want the agent to eventually be able to explain things about how it works. If you were to ask the agent any question, what would you ask it? Please list as many questions as you can think of.
- What challenges did you run into in interacting with the agent?
- What questions do you have about the system?
intro/info landing page
demographics survey
practice
novice - one of text or voice or voice/text
survey
novice - one of text or voice or voice/text
survey
novice - one of text or voice or voice/text
survey
survey compare all three
if advanced, then:
advanced - one of text or voice or voice/text
survey
advanced - one of text or voice or voice/text
survey
advanced - one of text or voice or voice/text
survey
survey compare all three
conclusion page