Project: Instructor Feedback Reports

[kariljordan]

Currently, instructors receive a SurveyMonkey link with their pre- and post-workshop survey results. This document, though useful, does not provide a way for instructors to understand the results of their surveys, or opportunities to improve if there scores are sub-optimal.

The instructor development committee has agreed to take on a project to create feedback reports for our instructors that will serve as a resource to help them improve (where needed) and congratulate them on their teaching.

I created a mock report in R. Keep in mind this mockup uses data from the trainer post workshop surveys. Once we have feedback reports for the instructors, we'd also like to create them for instructor trainers.

We'd like to use R Shiny.

We want 2-3 community members to help with this project.

Project Outline:

• [ ] Identify specifications to include in the post-workshop survey report for instructors.
• [ ] Create a ‘how to interpret your report’ document to send along with instructor feedback report.
• [ ] Create/revise a mock-up script for the Instructor Feedback Report.
• [ ] Set up automation for the script and report generation.
• [ ] Get feedback from the community about the look and feel of the report.
• [ ] Identify resources for coaching to supplement the report.
• [ ] Write a blog post to launch the reports.

Timeline

• This project will begin September 24th, and we'd like to have it complete by November 30th.

Action Required

• If you have thoughts about the project and/or ideas, please comment on this issue.
• If you are interested in actively participating in this project (contributing code, etc.), please e-mail Kari and comment below by September 24th.

[ha0ye]

I'd be interested in helping out as part of Hacktoberfest! LMK what to do.

[mfoos]

Hi all,

It's not clear to me how Shiny is adding to this project - it sound like the "deliverable" is static (personalized, but static) reports, rather than interactive webpages. Would the Shiny app be used by administrators to produce the reports?

[katrintirok]

I am interested to get involved, will think about it more in the next few days

[katrinleinweber]

I have another Shiny question/concern: I understood its usefulness for on-the-fly tasks (such as the instructor enters their workshop slug and gets the report generated). In that case, can you ensure that the Shiny code is cleanly be separated into a private, server part which has access to the raw data, and the public, client part? Please correct me if I'm branching off into a direction you weren't planning to go, anyway!

Since you also plan to "Set up automation for the script and report generation" (server-side, I presume), which task would actually be left for whoever should use the Shiny app in the end? I propose that it would be more consistent to always generate a report & mail it to the instructors automatically.

[kariljordan]

Let me start by saying I am not an expert, so if there is another package that we can use, I'm open to recommendations!

Now that there are a few people interested, I think it would be best if we had a short meeting where I can explain the entire workflow and what I'm trying to achieve. Then we can divide and conquer tasks, work asynchronously, and get this done!

Please add your availability here. Be sure to choose your timezone.

[kariljordan]

@maneesha would you mind completing the poll in the comment above with your availability? Or may @sheraaron? We need someone on the call that can talk us through the workflow of getting post-workshop results to instructors.

[maneesha]

Just filled in my availability - thanks for the ping. @sheraaron is welcome to join but I don't want to add to her workload.

[kariljordan]

We are meeting today (Sept 24) at 12:30pm EST. Click here for your timezone.

Here is the meeting location.

I sent calendar invitations for those I could find e-mails for.

Hope to speak with you all soon.

[kariljordan]

Notes from meeting:

• Can the CSV download have a link that can be reused in a script? First step is to figure out the SurveyMonkey API / and whether there's a public download of the CSVs available.
• The API requires a key: https://github.com/cloudyr/Rmonkey
• Kari will spend time figuring out how we can pull the data from SurveyMonkey easily so that the reports can be automatically updated when learners complete them.
• We need to remember that multiple instructors/helpers will receive the reports, so the responses from the reports are for the entire workshop.
• It would be nice to make these reports/dashboard cumulative so that instructors can see their scores over time, but we have to remember that the reports are for the entire instructional team, not only one instructor.
• Whatever we come up with, it needs to be easily manageable for workshop admins to use in their current workflow. Currently, they send a link to the instructional team 2-3 weeks before the workshop. They send one link for the pre-workshop survey responses, and one link for the post-workshop survey responses.

To Do List:

• [x] Everyone please have a look at the pre-workshop survey. What questions are pertinent for you to know in order to plan your workshop? Add your comments to the document.
• [x] Everyone please have a look at the post-workshop survey. What questions are helpful to you to understand how your workshop went? Add your comments to the document.

From there we will move to the tasks I outlined above:

• [x] Identify specifications to include in the pre/post-workshop survey report/dashboard for instructors.
• [ ] Create a ‘how to interpret your report’ document to send along with instructor feedback report.
• [ ] Create/revise a mock-up script for the Instructor Feedback Report.
• [ ] Set up automation for the script and report generation.
• [ ] Determine how workshop admins will incorporate this into their workflow.
• [ ] Get feedback from the community about the look and feel of the report.
• [ ] Identify resources for coaching to supplement the report.
• [ ] Write a blog post to launch the reports.

Lastly, I'd like to transfer this issue to the assessment repo. Does anyone know how to transfer an issue to a different repo in the same organization?

Thanks!

[katrinleinweber]

Transfering issues is not officially supported, but there are hacks.

However, please also consider the option to keep this one, but create new issues for specific tasks. With a link to this one, they can again be gathered in one place.

About public download of the CSVs available: It doesn't have to be public, just stable (or at least predictable) so it can be put into an automated script. Like we discussed recently in https://github.com/carpentries/assessment/pull/217/files#diff-0f1a91cb8b019aa1d0485338426f1c19 ;-) For that purpose, offline access was preferable, but the online access was such a stable link that we would need here.

[kariljordan]

Update: We are currently going through a reorganization of how we collect and store our survey data, so we do not need to worry about the SurveyMonkey/API issue. @fmichonneau will provide us with a "fake" dataset to use so that we can focus on the work of writing the code for these "live instructor feedback reports".

So, please check out this To Do List:

• [x] Everyone please have a look at the pre-workshop survey. What questions are pertinent for you to know in order to plan your workshop? Add your comments to the document.
• [x] Everyone please have a look at the post-workshop survey. What questions are helpful to you to understand how your workshop went? Add your comments to the document.

After that, check out the current source code for the reports from our pre and post workshop data. You can see the rendered report here.

I think a "next step" would be to discuss the look and feel of the future instructor report, specifically, are there better ways to show the plots and tables? Are there better ways to talk about/show the data in general?

For example, some people think rather than providing percentages, we could say something like "2 out of 5 people in your workshop felt X".

So, please have a look at the report, and keep in mind that for this report, the Data Carpentry Survey was different than the Software Carpentry Survey. We've since moved to a common survey among Data and Software Carpentry, and that survey is linked in the "to-do list" above.

P.S. Here is a folder with all of the archived and current surveys.

[katrinleinweber]

For completeness & posterity's sake here's a record of the requirement / user story as I understood it from the call:

a) Instructors and hosts want to receive an (as yet to be defined, see To Do List above) subset (or different subsets) of what the learner assessment reports contain. They want this around the time of the workshop, so that they can plan or assess it better.

b) That report must be update-able, to account for the fact that some learners will complete the survey later. It was stated that Shiny was desired because it might allow live-updating of a dynamic report. I suggest to re-evaluate this desire: How "live" might be sufficient for an instructor & host planning or debriefing. Maybe once per day or per week for about a month before or after the workshop?

Numbers on something like "completed survey submissions over time" would help nail down sensible timings for b). Does SurveyMonkey report something like this:

It's an example plot of podcast episode downloads during the week after publication.

[mfoos]

Hi all, sorry to do this, but I don't have the bandwidth to help out with this right now. If you think it would help, I'm happy to signal boost.

[kariljordan]

No worries @mfoos. Feel free to disable any notifications from this thread so you aren't bombarded. I received the data files from @fmichonneau and created a new repo under the carpentries organization so that we don't mess up anything in the assessment repo. @katrinleinweber @katrintirok and @ha0ye should have received invitations to join. I will add the data files, surveys, and code there and we can get to work.

@katrinleinweber your summary is spot on. SurveyMonkey does provide data trends like this:

[katrinleinweber]

Cool, thanks! Can the x-axis be transformed to something relative?

Regarding new repo: If anything from there needs to be merged back into assessment, please recreate the new repo as a fork of it (or of whichever repo this project's results need to feed back into). Alternatively: protecting master in that repo also prevents mess-ups ;-)

[kariljordan]

I spoke with @fmichonneau and we will keep these projects independent. I can protect the master.

[ha0ye]

I'm a little unclear on how the dynamic report generation will work, and I think that's a potentially limiting design decision. (and sorry if this has already been discussed or resolved - trying to record my thinking process here so that it might be helpful to others)

A Shiny app would need to run on a server, and that server would need to have the ability to retrieve the assessment data. I did see this post on stackoverflow about running shiny apps in GitHub pages, but that would mean that the ability to retrieve the assessment data is available to anyone who loads the website, which is problematic if there's sensitive information involved (which the SurveyMonkey API key is supposed to control access to).

We could build in an intermediate step where the data are retrieved from SurveyMonkey, anonymized, and then sent somewhere public, from which the reports are then generated. This would limit access to the full data to that intermediate step.

I've been thinking about continuous integration tools to do some of the automated report generation. The problem is, most of those are triggered by, e.g. updates to a repo. In the case of surveys, I imagine you would only be able to have a system that regularly tried to retrieve new data, or a system where one can manually refresh to set in motion the building of the automated report. Again, that feels like a case where a dedicated server to retrieve the data and build the reports might be necessary.

[fmichonneau]

When a learner fills out the pre- and post-workshop survey, it will trigger a webhook that will create a record in our self-hosted database. We have to do so because the data structure provided by surveymonkey is not stable and is not suitable for long-term archival of the data.

We will use redash to connect to this database and create queries that will generate anonymized datasets similar to the ones I provided to Kari. Access to the data will be protected by API keys. Redash caches the results of the queries and allows to run them at regular intervals.

The shiny app will take the workshop slug and a secret key (hash on the salted workshop slug), use the redash API keys to access the survey data, and generate the graphs/reports from there. We will run the shiny app on one of our servers.

We are working on putting together this infrastructure, and it's not something that this group needs to worry about.

Kari and the assessment team need help to come up with visualizations and statistics that are clear and provide (ideally actionable) insights to instructors.

[kariljordan]

Thank you @fmichonneau for your insight, because I had no idea how to describe that process!

So, as a reminder, here is what we've done so far, and what we are working on over on the instructor-feedback repo:

Complete

• [x] Look at the pre-workshop survey. What questions are pertinent for you to know in order to plan your workshop? Add your comments to the document.
• [x] Look at the post-workshop survey. What questions are helpful to you to understand how your workshop went? Add your comments to the document.
• [x] Identify specifications to include in the pre/post-workshop survey report/dashboard for instructors.

Upcoming

• [x] Create/revise a mock-up script for the Instructor Feedback Report.
• [x] Get feedback from the community about the look and feel of the report.

Mandatory for Kari, optional for others

• [x] Determine how workshop admins will incorporate this into their workflow.
• [x] Write a blog post to launch the reports.

Kari to complete with @fmichonneau

• [x] Set up automation for the script and report generation.

[kariljordan]

@katrintirok we are happy to announce that @fmichonneau is ready to release the instructor feedback reports! Your shiny app was EXTREMELY useful for us and gave us ideas to incorporate. We weren’t able to use the EXACT R code you created because of privacy concerns. The code for the app is in a private repo because there are a few sensitive things to make sure nobody can access the results. @fmichonneau would like to include you in the acknowledgements, so let us know if you're fine with that. THANK YOU!

[katrintirok]

Hi Kari, That sounds great, I am keen to see the new reports. I am fine with being included in the acknowledgements. Sorry for the late reply, unfortunately I don't have as much time for the carpentries at the moment. Best regards Katrin

On Thu, 12 Sep 2019, 17:28 Kari L. Jordan, PhD, notifications@github.com wrote:

@katrintirok https://github.com/katrintirok we are happy to announce that @fmichonneau https://github.com/fmichonneau is ready to release the instructor feedback reports! Your shiny app was EXTREMELY useful for us and gave us ideas to incorporate. We weren’t able to use the EXACT R code you created because of privacy concerns. The code for the app is in a private repo because there are a few sensitive things to make sure nobody can access the results. @fmichonneau https://github.com/fmichonneau would like to include you in the acknowledgements, so let us know if you're fine with that. THANK YOU!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/carpentries/instructor-feedback/issues/4?email_source=notifications&email_token=ACWK4NFMR53EGUODYWFPZ2LQJJNYBA5CNFSM4GCK2XN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6SJDNY#issuecomment-530878903, or mute the thread https://github.com/notifications/unsubscribe-auth/ACWK4NHN5W3WXXVCVEFNSUTQJJNYBANCNFSM4GCK2XNQ .