Brainstorm a plan for interviewing / collecting UX data from our users

[choldgraf]

In a recent meeting @sideye, Alan made a good point that we'll need to make a plan for how to gather information about the experience of the organizations that we're serving with these hubs. Let's use this issue to brainstorm the kinds of things we'd want to learn from them, as well as a plan for when to do checkpoints and interviews.

Resources

• Education hubs onboarding questionnaire
• Google Drive folder for storing interviews/research info

To Do

• [x] Create the initial onboarding questionnaire
• [ ] Decide what questions we want to learn from people using the infrastructure
• [ ] Create a mid-way checkpoint questionnaire

[sandeepsainath]

Hey Chris, here is a quick draft of my thoughts on what a plan for interviewing/collecting UX data from 2i2c users could look like:

1. Email campaigns

   • Similar to Adoption’s periodic emails (~1-2x/semester) to institutions asking them how their Data 8/JupyterHub efforts are going
   • Can be used to initiate interviews and/or fill out quick surveys (formatted below).

2. Surveys These surveys would be in the format of Google/Airtable forms with ratings fields (1-5) and short/long text fields.

   Ratings fields:

   • Overall experience
   • Ease of scalability (compared to previous infrastructure)
   • Costs (compared to previous infrastructure)

   • Current accessibility to support from 2i2c/DSEP

     Short/long text fields:

   • Benefits of 2i2c at their institution
   • General pain points in terms of:
     1. Infrastructure
     2. Pedagogical support
     3. Ease of onboarding 2i2c (maybe just for first survey/interview)
   • Other feedback/recommendations

3. Interviews

   • Can follow up with institutions part of an email campaign or follow ups to survey responses
   • Need to decide whether we should follow up with ALL institutions or only institutions with skewed survey responses (for ex. Several 1-3s for some categories, other categories are 4-5s)
   • Zoom calls hosted by either 2i2c/Adoption people for about 30-60 minutes

Timeline for collecting information (assuming 1 year-long pilot program):

We could aim for around 4 checkpoints (1 per “quarter" of the year) with additional check-ins for particular institutions with urgent issues.

This would exclude an additional end-of-program survey/evaluation which would involve a survey/interview focused on the institution’s reflection on the entire pilot program (and would thus be formatted differently then the above surveys/interviews meant for checkpoints).

Each checkpoint would involve an email campaign including a survey, and an interview to follow up. I anticipate our first few checkpoints will involve more interviews since institutions would be more likely to have issues related to things like onboarding.

[choldgraf]

Hey @sandeepsainath - I just realized that I didn't get back to you on this. A few quick thoughts:

• In general, this looks great and I really appreciate all of this structured planning. Agreed that we should plan on a couple different formats and touch-points depending on what we want to learn.
• Email campaigns
  • Do we have a list of contact info for the main points of contact for each of the hubs?
• Surveys + Interviews
  • Shall we (perhaps you, me, @ericvd-ucb at a minimum, and any others who are interested) have a meeting to determine: what we want to learn first / quickly, what we'll want to know on an ongoing basis, and what questions etc are best for answering each?
  • I wonder if running some preliminary interviews might be helpful in making some early changes to the service, pricing, etc. Perhaps we can start off with some in-person conversations, use this to guide early dev and service structure, and then begin the survey+follow-ups combo that you describe above.
  • We should define the kinds of users and what we want to learn from them (e.g., a student, an instructor, and an administrator all have useful information for us, but will have different kinds of information and require different kinds of questions)
• Communication channels
  • Currently we have a 2i2c Slack, GitHub issues, as well as any emailing lists people are a part of. Is this enough comms channels for now? Perhaps this is something we can probe in an early set of interviews etc.
  • Something else we could explore is a way to broadcast messages to hubs to send out surveys etc.

[choldgraf]

One more question for @ericvd-ucb and @sandeepsainath - any value in creating a DSEP-hubs-2020 channel in the 2i2c Slack for both team communication, as well as communication with any hub users that wish to join the Slack?

[sandeepsainath]

Hey @choldgraf,

• Email campaigns: Yes, there is currently a list of main points of contact for each hub. This has been shared in a previous email!
• Survey + Interviews: I like the idea of meeting up to discuss all the kinds of information we want to learn and track and from who, that will help consolidate this issue and brainstorm faster. Will reach out over email!
• Communication channels: I think the 2i2c Slack, GitHub issues, and email (lists or otherwise) are probably enough this early on pending ideas from early interviews/surveys. I particularly like Slack as a way to broadcast messages; having one, reliable, concentrated platform for discussion, general support, and advice could be great to facilitate a sense of community and Slack is probably the best way for everyone to stay in the loop.

A DSEP-hubs-2020 channel could be great just for communication between @ericvd-ucb, me, other DSEP members, and you! We're all very responsive on Slack. We could have one for hub users as well, but I personally think a dedicated, private channel just for us could be productive :)

[ericvd-ucb]

• Baseline Survey - would be great to get some sort of pre-or before survey going - to gauge how users are before running the pilot. One whole thing will to measure the confidence and competence of the professors/isntructors to see how that goes.

• I agree that a way to broadcast to partners would be userful. I would go with a google group email list for formality sake and completeness, and certainly a slack channel - but not sure what slack uptake percentage is ( high but not 100%)

[choldgraf]

I linked the onboarding questionnaire in the top comment, and also added a short to-do list to track subsequent items to discuss. I think right now we should decide what we want to learn from the folks that use the infrastructure.

@sandeepsainath do you have an idea for who is using the infrastructure now vs "sometime in the future"? I feel like this will influence our own timeline and checkpoints (e.g. doesn't make much sense to checkpoint a person that hasn't even started using the infrastructure!)

[sandeepsainath]

I think the most relevant factor to distinguish between "who" is using the infrastructure is experience with things like Jupyter, Python, and general sysadmin-like knowledge. In the short and "medium" run (so anywhere between now to 1-2 years), I expect that folks using the infrastructure as part of the pilot (and future pilots) will be professors who have a fairly limited knowledge base for the aforementioned tools.

These are professors who are likely new to at least one aspect of the infrastructure and will thus inevitably require support or some kind of guide to help with that. It's also worth mentioning though that some of these professors may have some course staff that may be more experienced with related tools. Regardless, I think the main target in gauging knowledge/experience should be the professors themselves.

@ericvd-ucb @kseniyausovich any thoughts to add or anything you disagree with based on your experience?

[choldgraf]

Maybe another way to phrase it is - what are we hoping to do with the information gathered from these surveys? Is the goal to improve DSEP support for them? Or for 2i2c to understand what is confusing about the infrastructure? Etc...

[kseniyausovich]

I’d say both.

On Thu, Feb 18, 2021 at 7:16 PM Chris Holdgraf notifications@github.com wrote:

Maybe another way to phrase it is - what are we hoping to do with the information gathered from these surveys? Is the goal to improve DSEP support for them? Or for 2i2c to understand what is confusing about the infrastructure? Etc...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/2i2c-org/pilot/issues/14#issuecomment-781785631, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE2DXUONM2G2XBGPJJB7A2LS7XJYNANCNFSM4SJETNAA .

-- Kseniya Usovich

B.A. Cognitive Science, University of California, Berkeley

[sandeepsainath]

Here is a first draft for a mid-way checkpoint questionnaire. Would love your thoughts on this @choldgraf and anyone else! The plan is to send this out sometime this week or the next depending on when we can finalize a draft.

[sandeepsainath]

Hey @choldgraf, just wanted to ask again if you could give a few suggestions or thoughts on the mid-way checkpoint questionnaire above. Thanks!