Feedback Learner Assessment Part I

[chendaniely]

Discussion issue for our first pass of survey items for our learners.

blog post here: http://software-carpentry.org/blog/2015/06/learner-assessment-part-01.html

[xuf12]

Thanks for the post Dan. Do we have a summary of current survey completion rate? I don't remember who is responsible for collecting or analyzing that data, but it'll help us iterate in the next round. My impression based on what I saw in the past is ~50% or less for the pre-workshop and lower for the post-workshop.

We tried to ask the participants to fill out the survey at the end of the workshop in Miami (posted a link on etherpad). I don't think it worked that well as they seemed exhausted and wanted to go home or to other commitments, so we didn't enforce it (not sure how either). I don't know the completion rates but I can ask.

I also have a few ideas after speaking with Greg. We all want to get the perfect survey with the highest response rate, but so far we only have one version and we can look at its performance from the data. Why don't we try a few different formats for both pre and post assessments, and pseudo-randomize them in different workshops to see which survey or method works best? Nothing will be perfect but at least we will get a feeling on what works better? The current pre survey takes about 5 min to fill out, and 10min for post. I feel that some answers like those typed-up ones aren't really analyzed, but they take up more time than MCQs. So maybe we can make a simplified, 1-min version of the pre survey and stick them to the registration page on Eventbrite? I have selected some questions and I can post them later. In any case, I feel that starting with the minimum amount of questions and iterate up may give better response results.

We may waste some data in this pilot, but like everything else, we are learning as we go. Experts will always say that there are flaws in the method, but I feel that we'd lose more if we don't try them.

What do you all think?

[raynamharris]

In the last paragraph of the blog, you state "But there are too many objectives in the lessons to list in a survey." Why not incorporate that into questions 2.1 of the Attitudes sections?

You don't have to ask about all the learning objectives, but you could ask about one beginner, intermediate, and advanced topic. For instance, instead of just asking if they were intimidated by "Shell" you could ask are you intimidated by "command-line interfaces", "for, while, and if loops", and "writing scripts from scratch". If you do that for git, R/Python, and SQL then Question 2 in the Attitudes section would have 12 questions instead of 4, but I think you would gain more information regarding how well students mastered the beginner, intermediate, and advanced topics.

[chendaniely]

@raynamharris

Below is a list of all the objectives for the shell lessons. There was an idea to scrape all the objectives from the lessons and have the students checkoff objectives they feel they have learned. we (@JasonJWilliamsNY, @drlabratory, and I) realized this will not be feasible for a survey.

shell objectives:

Explain how the shell relates to the keyboard, the screen, the operating system, and users’ programs. Explain when and why command-line interfaces should be used instead of graphical interfaces. Explain the similarities and differences between a file and a directory. Translate an absolute path into a relative path and vice versa. Construct absolute and relative paths that identify specific files and directories. Explain the steps in the shell’s read-run-print cycle. Identify the actual command, flags, and filenames in a command-line call. Demonstrate the use of tab completion, and explain its advantages. Create a directory hierarchy that matches a given diagram. Create files in that hierarchy using an editor or by copying and renaming existing files. Display the contents of a directory using the command line. Delete specified files and/or directories. Redirect a command’s output to a file. Process a file instead of keyboard input using redirection. Construct command pipelines with two or more stages. Explain what usually happens if a program or pipeline isn’t given any input to process. Explain Unix’s “small pieces, loosely joined” philosophy. Write a loop that applies one or more commands separately to each file in a set of files. Trace the values taken on by a loop variable during execution of the loop. Explain the difference between a variable’s name and its value. Explain why spaces and some punctuation characters shouldn’t be used in files’ names. Demonstrate how to see what commands have recently been executed. Re-run recently executed commands without retyping them. Write a shell script that runs a command or series of commands for a fixed set of files. Run a shell script from the command line. Write a shell script that operates on a set of files defined by the user on the command line. Create pipelines that include user-written shell scripts. Use grep to select lines from text files that match simple patterns. Use find to find files whose names match simple patterns. Use the output of one command as the command-line parameters to another command. Explain what is meant by “text” and “binary” files, and why many common tools don’t handle the latter well.

[jmxpearson]

Re: Attitude section rewording:

Why not phrase these in terms of comfort level? (e.g., "How comfortable were you with using each of the following prior to the workshop...")

Also, it's sometimes easier to assess an agreement with a statement, like

"I believe that Python will help save time in my workflow." Strongly Disagree <---> Strongly Agree

or

"I anticipate I will use git..." Daily, Every few days, Weekly, Monthly, Never

[gvwilson]

Can this one now be closed?