Questions to include in the instructor feedback surveys

[kariljordan]

Pre-Survey

• Please indicate your relevant fields or disciplines. Check all that apply.
• [ ] Agricultural or Environmental Sciences
• [ ] Biomedical or Health Sciences
• [ ] Chemistry
• [ ] Civil, Mechanical, Chemical, or Nuclear Engineering
• [ ] Computer Science or Electrical Engineering
• [ ] Earth Sciences
• [ ] Economics or Business
• [ ] Education
• [ ] Genetics, Genomics, or Bioinformatics
• [ ] High Performance Computing
• [ ] Humanities
• [ ] Library and Information Science
• [ ] Mathematics or Statistics
• [ ] Medicine
• [ ] Organismal Biology (Ecology, Botany, Zoology, Microbiology, etc.)
• [ ] Physical Sciences
• [ ] Planetary Sciences (Geology, Climatology, Oceanography, etc.)
• [ ] Psychology or Neuroscience
• [ ] Social Sciences
• [ ] Space Sciences
• [ ] Other ____

The concern is that there are a lot of choices. Can we only include the top 3?

• What is your current occupation/career stage? Check all that apply.
• [ ] Administrative Staff
• [ ] Analyst
• [ ] Commercial Software Developer
• [ ] Faculty
• [ ] Government Employee
• [ ] Graduate Student
• [ ] Industry Employee
• [ ] Librarian or Archivist
• [ ] Management
• [ ] Postdoctoral Researcher
• [ ] Research Staff (including Researcher Programmer)
• [ ] Retired or Not Employed
• [ ] Support Staff (including Technical Support)
• [ ] Undergraduate Student
• [ ] Other (please specify) ____

The concern is that there are a lot of choices. Can we only include the top 3?

• What operating system is on the computer you are bringing to the workshop?
• [ ] Apple/Mac OS/UNIX
• [ ] Linux
• [ ] Windows
• [ ] Not sure
• How often do you currently use any of the following?

	Never	Less than once per year	Several times per year	Monthly	Weekly	Daily
Statistical analysis programs with a graphical user interface (SPSS, SAS, etc.)
Programming languages (R, Python, etc.)
Databases (SQL, Access, etc.)
Version control software (Git, Subversion (SVN), Mercurial, etc.)
A command shell (usually accessed through Terminal on Mac OS X or PowerShell on Windows)

• Why are you attending this workshop? Check all that apply.

• [ ] To learn new skills.

• [ ] To refresh or review my skills.

• [ ] To learn skills that I can apply to my current work.

• [ ] To learn skills that I can apply to my work in the future.

• [ ] To learn skills that will help me get a job or a promotion.

• [ ] As a requirement for my program or current position.

• [ ] Other: __

• Please share what you most hope to learn from attending this workshop.

========================================================================== Post-Survey

• Please rate your level of agreement with the following statements.

	Strongly Disagree	Disagree	Neither Agree nor Disagree	Agree	Strongly Agree
Having access to the original, raw data is important to be able to repeat an analysis.
I can write a small program, script, or macro to address a problem in my own work.
I know how to search for answers to my technical questions online.
While working on a programming project, if I got stuck, I can find ways of overcoming the problem.
I am confident in my ability to make use of programming software to work with data.
Using a programming language (like R or Python) can make my analyses easier to reproduce.
Using a programming language (like R or Python) can make me more efficient at working with data.

Show the pre-post comparison

• I felt comfortable learning in this workshop environment. [ ] Strongly disagree [ ] Disagree [ ] Neutral [ ] Agree [ ] Strongly Agree

• I can immediately apply what I learned at this workshop. [ ] Strongly disagree [ ] Disagree [ ] Neutral [ ] Agree [ ] Strongly Agree

• Please list the major strengths of this workshop.

• Please list the ways the workshop could be improved.

• Were there any accessibility issues that affected your ability to participate in this workshop?

• [ ] No

• [ ] Yes

• If you answered yes to the question above, please describe what the issues were.

• Please rate your level of agreement with the following statements.

	Strongly Disagree	Disagree	Neither Agree nor Disagree	Agree	Strongly Agree
I was able to get clear answers to my questions from the instructors.
The instructors were enthusiastic about the workshop.
I felt comfortable interacting with the instructors.
The instructors were knowledgeable about the material being taught.

• Please provide an example of how an instructor or helper affected your learning experience.
• How likely are you to recommend this workshop to a friend or colleague?

========================================================================= Now that we know which questions to include, we are ready to create the script using this data. Please keep in mind we should make it as reproducible as possible.

There should be two separate scripts written. One for the pre-workshop and one for the post-workshop, but they should have the same overall look and feel. In the code folder you'll find the current report that includes all workshop survey data, and a mock-up report that Kari started a while ago.

Please add your suggestions below for how to proceed with generating the code.

[ErinBecker]

This looks awesome @kariljordan. When would you like feedback by?

[kariljordan]

Once the code is generated we will send it around for review for look and feel. Thanks @ErinBecker!

[kariljordan]

@fmichonneau created a preliminary script to give us an idea of options and to also inspire us to push this project forward. You can see the rendered script here. It takes a while to load FYI, and keep in mind there's only one response to the survey.

We can either use this infrastructure and make changes to the visualizations/summary statistics, or we can start from scratch.

Please share below what you'd like to do @ha0ye @katrinleinweber @katrintirok

[ha0ye]

I'm totally swamped with things for the next week or so. I would like to contribute though, so if you have identified specific tasks that I can help out (and just assign one to me), I can get around to it maybe next Thursday.

[katrintirok]

just looked at Francois's report draft, I would be fine starting from there, especially making use of the data import. However I also liked the mock report you had started. I also had a look at the feedback report, there are some nice graphs too, I particularly liked the one that compares the pre- and post-workshop skills.

A few questions:

• do we want to use the same type of visualisation for each question or a few different ones?, I would like to have a bit variety, not too much though
• I agree that pre- and post workshop report should have the same look, I like how Francois made the 2 tabs for both
• how would we go about splitting up the work? use the data import as in fmichenneau's script and then each of us working on different questions of the survey? Then review each others code with specific rules in mind?

[kariljordan]

We should definitely include the paired plot, but not everyone completes both the pre and post workshop survey, so we'll just need to figure out a plan if n is too small.

I like having a variety of visualizations as well.

I think it would be great to use François' script and take it from there. @katrintirok would you like first dibs on which questions you'd like to work on? I like the plan you've outlined in the third bullet:)

[katrintirok]

Great

@kariljordan could you add François' script to the code folder (or did I miss it somewhere?).

I would start with the first 3 questions of the pre-survey we said to include, i.e. questions 5., 6. and 9. in the commented version.

[fmichonneau]

Hi @katrintirok

My parental leave started before I could make the code for the Shiny app publicly available. There is a little more clean up that I need to do before opening it up.

That being said, most of the code to generate the graphs is taken from what I had written for the report here: https://github.com/carpentries/assessment/blob/master/learner-assessment/reports-src/2018-07-learners-pre-post-surveys.Rmd

Don't worry about making the code work with the app for now. I'll easily be able to adapt it and plug it in the app.

We are looking for creative and informative visualizations based on the answers, and text that instructors can use to interpret these graphs.

A few questions that came to mind as I started working on this:

• how many pre- and post- data do we need to show the paired plot (using the learner identifier to match the answers or even just compare the results for the same questions in the pre and post surveys)?
• how to report/visualize answers that learners have skipped?
• would using the sidenotes of the Tufte template (https://rstudio.github.io/tufte/) be a useful way to display notes on how to interpret the graphs?

[kariljordan]

Thank you François for responding to this issue while you're on leave. Please enjoy the new baby!

@katrintirok are you comfortable with using the current code and working on 5., 6. and 9 in the pre-survey?

[katrintirok]

Yes, thanks for the info and the link to the tufte documentation.

I will focus on visualisations and text and will check with the current code. I will work on 5., 6., 9. as a start and then check again

[kariljordan]

Awesome! Do you want to work it where you go first, and then pass along the code to the next person? Or should others work on questions simultaneously?

[katrintirok]

I think we need to work simultaneously on different questions to meet deadlines, if I have some code I will share, should we work on different branches in this repository, what would be the best way?

[kariljordan]

Yes let's work on different branches. @katrinleinweber are you able to work on a few questions?

[shippy]

Are there any visualizations that aren't being worked on but should be? (Or is it too late to join in?)

[kariljordan]

@shippy it's not too late!

Are you able to look at questions 10 and 11 for the pre-survey? The questions are:

How often do you currently use any of the following?

	Never	Less than once per year	Several times per year	Monthly	Weekly	Daily
Statistical analysis programs with a graphical user interface (SPSS, SAS, etc.)
Programming languages (R, Python, etc.)
Databases (SQL, Access, etc.)
Version control software (Git, Subversion (SVN), Mercurial, etc.)
A command shell (usually accessed through Terminal on Mac OS X or PowerShell on Windows)

Why are you attending this workshop? Check all that apply.

• To learn new skills.
• To refresh or review my skills.
• To learn skills that I can apply to my current work.
• To learn skills that I can apply to my work in the future.
• To learn skills that will help me get a job or a promotion.
• As a requirement for my program or current position.
• Other: __

[kariljordan]

Hello @katrinleinweber @katrintirok @ha0ye @shippy I wanted to check in on our progress. I think we are still on track to have the code finished by November 30th, but wanted to make sure all of the questions are being worked on.

[katrintirok]

Hi @kariljordan @ha0ye @shippy @katrinleinweber I just pushed a branch (katrin) I am working on, which has a notebook katrin_qu1to3.Rmd with what I have done so far, please have a look, a few things still to do - generalise data import, put some code into functions (esp for questions 2 & 3)

[kariljordan]

@katrinleinweber I'd like to review your work and also share it with the assessment team on staff. Do you have gh pages set up on your branch? I'd like to be able to click on the HTML file and see the report. My git skills are not perfect, so when I tried cloning the report and opening the file from your branch in RStudio it didn't work. I've been trying for a few days, that's why it's taken me so long to respond.

[katrinleinweber]

Is there is a mix-up here? Do you mean @katrintirok's branch? I didn't reply on Oct. 10th and couldn't follow this further, I'm sorry. I guess it's best to https://github.com/carpentries/instructor-feedback/pull/5.

[kariljordan]

I am so sorry! I meant @katrintirok's branch.

[katrintirok]

@kariljordan Hi, sorry that you can't see, I have not set up GitHub pages on 'my' branch (not sure how I can do that), I had added an html notebook, but I guess it would only show when downloaded, if you have a suggestion what to do to make it easier visible online. It is not more than a first try of how to visualise those questions yet ... I started it as a notebook now, but remembered the original idea of using shiny. However, I got a bit confused - I understood that François would do the automation setup on web and also automation of data import ... ? Would we still use shiny then? I am away until 27th November, but could work on it again after that, although only little bit at a time

[kariljordan]

@katrintirok please see comments in #4. We appreciate your hard work and knowledge, and would like to include you in the acknowledgments when we release the feedback reports!