TWE: Create Research Plan for RP017

[bradyse]

Dependencies

• [x] #309
• [x] A research setup structure (wiki, GitHub labels, Google Drive, etc.) has been created. #291

  Overview

  We need to create a Research Plan for Productivity Dashboard Interviews so that we can understand interns' experience with the productivity tracker.

Action Items

• [x] Update this issue to make sure it's properly categorized and easy to manage
  • [x] Under "Projects," add to the Project Management Board (helps with Project Management overview)
  • [x] Add the milestone: Research Plan Creation (helps with prioritization)
  • [x] Add label of the correct research plan number associated with this research plan (e.g., Research: RP001) (helps you find related issues in the same workflow)
  • [x] Edit title of issue with the correct research plan number (same as prior)
  • [x] Edit placeholders in Overview of this issue with the relevant information in square brackets. (personalizes the issue to make it easier to read). Any place you see [replace...] replace both the text and the brackets with what it is asking for (including in the front matter above but not where it says name, but not where it says title.
• [x] Update this issue with the relevant Resources:
  • [x] Go to the Research Output Overview Page (link in Resources)
    • [x] find the relevant overview type wiki page.
    • [x] change [Wiki [TYPE] Research Overview Page] in the Resources below with the correct wiki page of the research overview by type.
  • [x] Go to the Research Output Overview Page again
    • [x] find the relevant wiki page for this specific research plan.
    • [x] change the [RP__ Wiki Page] in the Resources below with this RP's specific wiki page.
  • [x] Find and link the folder you are going to put your research plan in (e.g., Research by Type>2. Intern>RP6 - Intern Intake Interviews)
    • [x] Go to the Google Drive's Research by Type Folder (link in Resources)
    • [x] Go into the relevant Type Folder (e.g., 1. Mentor, 2. Intern, etc.)
    • [x] once inside, right-click on the relevant destination folder and copy the link. (e.g., RP6 - Intern Intake Interviews)
      • [x] Add the Google Drive link in the Resources below with the link you just copied and revise the display text to include the Research Plan number (e.g., Google Drive RP__ Folder to Google Drive RP008 Folder)
• [x] Create the Research Plan Document.
  • [x] Make a new document using the TWE Research Plan template (link in Resources). Follow the instructions in the How To Write a Research Plan guide (link in Resources) and mark your progress in this issue.
  • [x] Move your document from "My Drive" to the shared drive internship folder of this specific RP.
  • [x] Name your document to be called "TWE: RP___ : [TITLE OF RESEARCH] Research Plan".
  • [x] Update the document with the relevant information in accordance with the research roadmap document (found on the Research Overview Page).
    • [x] Clean up transcript
    • [x] Move de-identified transcript into RP project's Google Drive folder
    • [x] Document all questions and follow-up questions by interviewer
    • [x] Identify protocol based upon research questions
    • [x] Summarize transcript for basic themes
    • [x] Write data analysis for identification of basic themes
  • [x] Update the Table of contents (page 2) when you are finished with the document.
• [x] Meet and sign-off research plan document with UX Research lead (by adding it to the research team meeting agenda).
• [x] Update RP Wiki Page with any changes or corrections.
• [x] Once research lead has signed off, add it to the PM/Research lead agenda
• [ ] sign-off by PM
• [ ] Does this issue have any dependency checkboxes in the action items here?
  • [ ] if yes, remove dependencies that are listed below (issues that are waiting on this issue to be completed. Sometimes they have already been made and sometimes they yet to be made).
    • [ ] close this issue
  • [ ] if no, check to see if the follow-up issues have already been made (see related structure issue). If new issues have not been made, then apply label ready for research lead and define what needs to be done in a comment below
    • [ ] close the issue
    • [ ] move to Question/Review Column

Resources

Resources needed to complete issue

• Wiki Research Output Overview Page
• Google Drive Research by Type Folder
• TWE: Research Plan Template
• TWE: How to Write a Research Plan

  Resources gathered during the completion of the issue

• Wiki Intern Research Overview Page
• RP017 Wiki Page
• Google Drive RP017 Folder
• RP017 Research Plan
• RP017 De-identified Transcript

[bradyse]

Note that research plan needs to be adjusted based upon one interview. See comment (and decision) made here.

[bradyse]

2023-06-27 - Provide Update

1. Progress: RP is almost complete.
2. Blockers: It was hard to write the KPIs for this RP since the data was already collected.
3. Availability: I will have time this weekend to work on it (1-2 hours)
4. ETA: Next week
5. Pictures (if necessary):

[pandanista]

We don't have defined KPIs listed for the internship, but we have a couple of examples. The KPI examples can be found in

• RP001's research plan, which has listed KPIs approved by product.
• RP002's research plan also has KPIs listed.

Hope this helps with the KPIs.

Since we did one interview, I think we could align the product goals based on the interview transcript.

[bradyse]

1. Progress: In process of cleaning up original transcript. It is not fully deidentified yet. Because of this, it's still in the "My Drive" of the internship account. 2. Blockers: No blockers, but the transcript is slower going than I thought, because Bonnie didn't turn on the transcription until midway through the interview. So I am transcribing it by hand. 3. Availability: 1-2 hours 4. ETA: 2 weeks 5. Pictures or links (if necessary):

* remember to add links to the top of the issue if they are going to be needed again.

[pandanista]

1. Progress: In process of cleaning up original transcript. It is not fully deidentified yet. Because of this, it's still in the "My Drive" of the internship account. 2. Blockers: No blockers, but the transcript is slower going than I thought, because Bonnie didn't turn on the transcription until midway through the interview. So I am transcribing it by hand. 3. Availability: 1-2 hours 4. ETA: 2 weeks 5. Pictures or links (if necessary):

* remember to add links to the top of the issue if they are going to be needed again.

Thank you for the update, @bradyse. Transcribing def. takes time since we don't have a paid otter.ai account. But Otter also allows 3 free transcripts lifetime if you register for a personal account, which might be worth it.

[bradyse]

Thank you for the update, @bradyse. Transcribing def. takes time since we don't have a paid otter.ai account. But Otter also allows 3 free transcripts lifetime if you register for a personal account, which might be worth it.

1. Progress: Made a lot of progress. I don't think I need Otter, as I'm now cleaning up the Zoom transcript.
2. Blockers: No blockers but it just is taking a bit more time. Will try to work on it while on the road traveling this weekend.
3. Availability: 1-2 hours
4. ETA: 1 week
5. Pictures or links (if necessary): Link to in-progress transcript.

[pandanista]

@bradyse I just had a thought, and I wonder if you could help.

Are you doing de-identification after or along the transcript cleaning up process?

If yes, would you please write down the steps you took to de-identify the participant? So I could compare to what we have on the WIP transcribe template and incorporate some changes if needed.

Let me know what you think. Thank you!

[bradyse]

Are you doing de-identification after or along the transcript cleaning up process?

If yes, would you please write down the steps you took to de-identify the participant? So I could compare to what we have on the WIP transcribe template and incorporate some changes if needed.

Hi @pandanista, I just finished cleaning up the transcript and moved the de-identified copy into this RP's research folder. Here were the main steps I took:

1. Create a branded document with the raw transcript copied and pasted into the document.
2. Save this file inside the PII folder along with the video or other transcript files.
3. Clean up transcript:
   • Separate text according to speakers and timestamps (timestamps should match the video file timestamps so that a researcher needs to go back and double check the original video)
   • Add missed words
   • Change misidentified words
   • Deleted repeated words
   • Remove filler words such as "um" and "uh"
   • Where needed, add context in square brackets so that a reader could understand what was happening without watching the video (e.g., [steps off camera])
4. De-identify the research participant or any names mentioned between the interviewee and interviewer. (Note, I did not de-identify the interviewer)
   • Replace the interviewee's name inside the transcript with the participant type (e.g., "Intern")
   • Replace mentioned people in the transcript according to their roles inside square brackets so the reader knows this information was redacted (e.g., "[team member]".
   • If the folder where the video is located has the interviewee's name, rename the folder to replace their name with the participant type
5. Add relevant details to the transcript file regarding the date of interview and include the link to where the original video is.
6. When you're satisfied that the transcript is completely deidentified:
   • Click on the version history and select the most current version.
   • Click the three dots and select "Make a copy"
   • Save a copy into the shared drive folder where you want to save the de-identified version. This is to make sure that the version history does not include any identifiable information.
   • In the copied file, rename the file name to remove the "Copy of" text and the time stamp text.

[pandanista]

Thank you for sharing the progress and the steps, @bradyse.

[bradyse]

@pandanista

1. Progress: The RP017 Research Plan is ready for review by UX Lead and Product.
2. Blockers: None
3. Availability: 1-2 hours/week
4. ETA: I can meet next week as needed
5. Pictures or links (if necessary): RP017 Research Plan

[pandanista]

@bradyse Thank you very much for all your efforts! Has Bonnie provided any links to TDM usability tests yet?

@adarosh As a follow-up, please use the action items listed below as a to-do list for RP017's peer review.

Action Items for Peer Review

• [x] Review this issue
  • [x] Read the issue overview to learn about why this document was created.
  • [x] Locate draft of the document or spreadsheet in the Resources.
• [x] Review draft
  • [x] Preview the draft quickly to get a sense of how the draft is structured and the type of content.
  • [x] Review the structure of the draft to ensure that the headings (in a Google Doc) or columns (Google Sheet) are clearly identified.
  • [x] Determine if the information is communicated in a way that is concise, precise, and free from jargon.
  • [x] Identify whether the draft adheres to the basic branding guidelines for the project (see Branding or Design wiki link in Resources).
• [x] Provide feedback
  • [x] Add feedback in the form of (1) questions and/or (2) explicit instructions for how the content should be improved. Feedback in a Google Doc should be given in the form of comments.
  • [x] If there are edits to be made, please use "suggesting mode" in Google doc instead of "editing mode".
  • [x] Determine the accuracy of the content for each section (Google doc) or cell ( Google sheet). If you find an inaccuracy, error, or mis-categorization, explain how the information presented could be improved.
  • [x] For each suggested improvement, provide a link or reference within your comment to justify your suggestion (if available).
• [x] Specific to this issue:
  • [x] Review documents linked in the comments in this issue to gather more information
  • [x] Review Wiki Intern Research Overview Page
  • [x] Review RP017 Wiki Page
  • [x] Log into the Internship@hackforla.org Google account and go to drive.google.com under that account
  • [x] Go to shared drives > Internship - PII > Internship - PII > TWE Interview Recordings by Plan # > RP017 folder > recording with Intern about the dashboard 2022-07-13 folder
  • [x] Watch the interview at 1.5X or 2X speed
  • [x] Review the de-identified transcript to see if it needs any editing to provide more context or clarifications and/or redact any information sensitive
  • [x] Review TDM's usability testing example (TDM Calculator has done usability tests that might be relevant to the productivity tracker interview)
  • [x] Please provide concrete feedback based on the reviews you have done above in the Research Plan draft.
• [x] Once all the steps have been completed, reach out to the original owner of this issue to determine if they can complete the next steps in the issue.

Thank you both!

Edited: Corrected the project name from VRMS to TDM. Added the TDM usability example and project card URL to the instruction section above.

[bradyse]

@bradyse Thank you very much for all your efforts! Has Bonnie provided any links to VRMS usability tests yet?

@pandanista No, she hasn't provided any links to my knowledge. I was a bit unsure what exactly she wanted me to do with the Transcripts but I do know adding screenshots would be helpful. It also would be just as helpful if whoever was doing the analysis just had access to the recording itself.

[pandanista]

@bradyse Thank you very much for all your efforts! Has Bonnie provided any links to VRMS usability tests yet?

@pandanista No, she hasn't provided any links to my knowledge. I was a bit unsure what exactly she wanted me to do with the Transcripts but I do know adding screenshots would be helpful. It also would be just as helpful if whoever was doing the analysis just had access to the recording itself.

Thank you so much, Sara. Keep us posted if there is any update on the VRMS usability tests. Otherwise I can check in with Bonnie next week when she is back. I will also check in with Angela to make sure she has access to the recording file.

[bradyse]

1. Progress: None
2. Blockers: Reached out to Ebi on Slack
3. Availability: 2-3 hours
4. ETA: Unsure
5. Pictures or links (if necessary):

[pandanista]

Hi @bradyse,

Thank you so much for your efforts on putting RP017 together!

I have a couple of questions regrading the next steps:

• What do you think is the best approach to move RP017 forward given the current status?
• What are the remaining steps in finalizing this research plan?

Also, thank you for taking the time to review the current materials, @adarosh. :)

[pandanista]

@adarosh Would you please provide update on the peer review? Thank you.

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[adarosh]

@bradyse @pandanista

Progress:

• Completed peer-review.
• Confirmed that all hyperlinks under Table of Contents work.
• Made some suggested minor edits for clarity and grammar.
• Added feedback on content. Below I have listed the major feedback for consideration (this feedback is also listed throughout the document). All edits/comments were made in Suggesting Mode.

Feedback:

• Noticed that the branding on the bottom of the first page of the document says, “Hack for LA is a brigade of Code for America.” Just checking (since the brigades became independent)- is this the most current branding?

• There are a lot of hyperlinks under Table of Contents, some of which all link to the same page (e.g., Participants, Research Assumption, Research Method: Interview, Purpose of Method, Recruitment, Protocol all link to the same page). For a brief document, you might consider condensing the number of hyperlinks to major sections.

• Prior to Protocol section on pg 5 (besides on the title page), the product is called the "productivity tracker." Recommend deciding between calling it a tracker or dashboard, and keeping the term consistent throughout the title/document.

• Similar to comment above, under Protocol section, the same spreadsheet is referred to as “skill tracking spreadsheet” and then “skill types spreadsheet.” Will want to make sure terms are consistent throughout document.

• Consider adding link to skill tracking spreadsheet (referred to in Protocol section) and Productivity Tracker under Resources section in document.

Re: the Data Analysis plan:

• Not sure it would be considered RQA with a sample size of one- would consult Linda K. to see if that would be applicable. If using RQA, could add a few things to the Data Analysis section (per RQA guide and Hamilton’s slide deck): identify type of analysis (thematic, content, narrative), note coding type (internal/inductive vs. external/deductive), and state which domain name corresponds with each question (since in RQA the neutral domain names should line up with the interview guide).

• It might also be helpful in the Data Analysis section to note that, in addition to creating summaries based on the domains, team members will want to add any important information that does not fit into the domains to an “Other Observations” section.

• Once the team members conducting the analysis are identified, it might be helpful to hold a training session where everyone reviews the RQA guide/Hamilton’s slide deck.

Hope this helps. Feel free to reach out via slack if you have any questions about the feedback.

Blockers: None Availability: N/A ETA: N/A Pictures or links* (if necessary): N/A

[pandanista]

@adarosh Appreciate all the feedback! I will review everything, and we can go over them in details in our next UXR meeting o Tuesday.

[pandanista]

@adarosh and @bradyse: Thank you both for working on RP017's research plan!

We reviewed the research plan, and addressed @adarosh's feedback at the UXR meeting on 9/26. There are still some comments left on the document, and I will get Project Lead's feedback on those before making more revisions.

Will keep you both updated.

[pandanista]

Let's use case study to analyze the data. @adarosh Please incorporate the new research method into the research plan. Thank you.

[pandanista]

Also, since the skill set spreadsheet includes PII and will be challenging to de-identify since GitHub issues and the intern's GitHub handle are included, we decided not to include the sheet in the research plan document. Let me know if it would present any difficulty for the data analysis stage.

Right now, the dashboard identifies interns by number (the intern numbers are not consistent with IS21's interview roll call and session table because it was done before we assigned interns numbers retroactively). Each intern's information can be viewed by the others.

In the future, each intern will only be able to view their own individualized dashboard and aggregated dashboard for the whole group to protect privacy.

[pandanista]

@adarosh Thank you for agreeing to implement the changes to RP017's research plan.

I created a folder called Propsed research methods in the RP017 folder. We can download and upload some literature that is useful to our decision-making on the choice of method into the folder. Looks like both qualitative usability testing and a case study will be delivered as a report. The only concern I have is we only have a sample of one. Not sure if that would be sufficient to justify using the qualitative usability test as the method. Let me know your thoughts.

A few resources for case study as a research method I found: Case Study Methodology of Qualitative Research: Key Attributes and Navigating the Conundrums in Its Application; Case Study Method: A Step-by-Step Guide for Business Researchers; The case study approach Methodology or method? A critical review of qualitative case study reports UCSF libraries case studies research guide

Qualitative usability testing resources and examples (it seems hard to find published literature on that): Qualitative Usability Testing: Study Guide; usability.gov's usability test results reporting; Usability guide by Maze; Usability Testing Report by U of M Dept of Homeland Security's example

[adarosh]

Progress: Went through document and made language consistent by changing mentions of tracker/dashboard to intern activity dashboard. Updated data analysis section with planned method of case study. Timeline will need to be updated. Blockers: Changes throughout document and updated data analysis section are ready for review. Availability: N/A ETA: N/A Pictures or links* (if necessary): N/A

[pandanista]

Thank you, @adarosh, for the update.

Next:

• Link consent form document. I found the consent from their parents, see the parental consent responses
• Make a copy of the spreadsheet and redact the content so nothing will be identifiable. Then save the screenshot(s) in RP017's folder so other team members will have more context when they review the interview transcript.
• UX Lead will create a new issue to track data analysis.

[pandanista]

@adarosh I have created #492 and #493 for RP017's data analysis. Let me know when you are done with the screenshots of the spreadsheet. Thank you.

[pandanista]

@adarosh I have created #492 and #493 for RP017's data analysis. Bonnie suggested another way to redact which sounds better because we can interact with the spreadsheet copy.

1. Make a copy of the existing spreadsheet
2. Mark all the column content black so PII doesn't show
3. Make a copy of the latest version of the Google sheet so no one can see the editing history
4. Update the column cells to blank
5. Save it into RP017's folder

That way, we can still interact with the spreadsheet rather than a screenshot. Let me know if you have questions. Thank you.

[sunannie27]

Please provide update

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[pandanista]

@adarosh Please provide update:

1. Progress: "What is the current status of your project? What have you completed and what is left to do?" - we know you just came back from break, please just summarize the progress from before then
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

Thank you.

[adarosh]

Progress: Made copy of Intern Activity Spreadsheet and saved to RP017 folder in Internship drive. Redacted identifiable information.

Under Resources section on this issue- RP017 Wiki Page link goes to a blank page.

Blockers: Copy of Intern Activity Spreadsheet is ready for review by UX Lead to confirm that all identifiable information has been redacted.

When last discussed, it was not yet determined which team member will be responsible for case study.

Availability: N/A

ETA: N/A

Pictures or links* (if necessary): N/A

[pandanista]

Thank you for the update, @adarosh.

I've updated RP017's wiki page link in the research plan and above.

Bonnie and I reviewed the Intern Activity Spreadsheet on 11/29's UX lead/PM meeting, and the de-identified spreadsheet copy is approved. I've updated the links in the research plan.

We also have agreed not to include the KPIs for this research plan for now since we are in the process of identifying KPIs for the TWE project. We will include RP017's KPIs at a later stage.

@sunannie27 RP017 Research Plan is ready for product to review and added to all team meeting's agenda. Meanwhile, let us know if you have any questions.

[ExperimentsInHonesty]

I added a comment to the document instead of doing a suggested text, because I want the person writing this document to know the full context but want you to figure how to add the information and contextualize is relevance.

This intern was also a returning intern. This was their second year. Additionally, this intern was an outstanding performer. Doing more issues than any other intern in the first cohort.

[sunannie27]

@adarosh Bonnie and I reviewed the Research Plan and left comments for you.

[pandanista]

@adarosh Hi Angela, the PM and the project lead provided feedback for RP017 research plan. Please take a look at the comments and see how to incorporate them into the research plan document. Let us know if you have any questions.

[sunannie27]

@adarosh Please provide update

1. Progress: "What is the current status of your project? What have you completed and what is left to do?"
2. Blockers: "Difficulties or errors encountered."
3. Availability: "How much time will you have this week to work on this issue?"
4. ETA: "When do you expect this issue to be completed?"
5. Pictures or links* (if necessary): "Add any pictures or links that will help illustrate what you are working on."

• remember to add links to the top of the issue if they are going to be needed again.

[adarosh]

@pandanista @sunannie27 - The team lead Ying and I went through the document together last week (1/30) to address comments. For the few comments that needed clarification, Ying will discuss with team before we finalize.

[pandanista]

@sunannie27 We have addressed all the comments and made edits in the RP017 Research Plan. Please let us know if you have any other questions before the final approval.

[ExperimentsInHonesty]

@adarosh I have reviewed RP017 Research Plan and made a few grammar changes and suggestions.

Additionally, the sanitized version of the spreadsheet is linked to the research plan, but it would be good if we linked the actual one with a note saying it has restricted access because it contains pii

[adarosh]

Discussed research plan in UXR meeting on 3/12. Reviewed and made minor edit to paragraph addition in Background section. @pandanista will add action item spreadsheet.

[pandanista]

• Added the following assets to the resources section of the research plan document.

  • The action steps spreadsheet template
  • The action steps spreadsheet examples
  • The case study report template
  • The OG intern activity spreadsheet

• Made a note on the intern activity spreadsheet linked in the research plan body is de-identified, and the OG version is saved on the PII drive

• A new timeline with start dates and duration for future research is added.

@sunannie27 We are ready to close this issue since we have addressed all the comments left by Product. Please see the updated research plan document. Let me know if there is anything else.

[sunannie27]

@pandanista Looks good to me.

[ExperimentsInHonesty]

@adarosh Thank you for the work on this. We are closing this issue.