Self-assessment test for “advanced” Carpentry workshops

[arockenberger]

There is currently high demand for non-beginner workshops in Python, R and other Carpentries lessons but we have not yet had the capacities to meet those needs of our learners. Another issue is that our learners under- or overestimate their knowledge and understanding of core concepts of the material we are teaching. I had an idea as to how we could try to address this: With a short self-assessment test that can be done before signing up for a workshop. I thought about using exercises from SWC materials. I don't think such a test should be mandatory, but I guess it will be appreciated by most. Something along the lines of:

"Is this course for me? Assess yourself with these short exercises and find out if the level of this workshop matches your knowledge and understanding of core concepts." Depending on how advanced the workshop is, 3-10 questions. Give suggestions for level placement for low scores: “Even though this course might not be the right fit for you, we have xxx that matches your experience!”

What do you think?

[annesbe]

I think this could be very useful! Even though we have a course description many don't read it, and the description does not always necessarily tell someone if they might find the workshop useful or not, because that might require a great deal of detail.

[drmowinckels]

I agree. I think this is an excellent idea. And it really depends on the course.

For instance for SWC R and the tidyverse R there is a clear difference with who might benefit, and giving people a small set of questions that might help them evaluate if the course is of value to them is a very nice way of guiding people to the right decision.

[desireetreichler]

I also agree! On longer run, it would be nice to do this in a way that is shareable with the Carpentries' community, as other host institutions may have the same problem - and such questionnaires would be lesson-specific and reusable each time the workshop is run. Maybe useful to have this in the back of our heads already now when choosing where and how to host the self-check questionnaires/exercises.

[Teebusch]

There must be websites that allow us to do this thing automatically. I know Repl.it (the site I used for the AR workshop) lets you build automatically graded programming exercises. I dont know what the limits of the free model are. I will have a look.

[lexnederbragt]

I like the idea. However, it could become a burden if we were to ask people to perform programming tasks, these take time and may feel like an exam. How about posing a programming (or shell or version control problem) and letting them self-evaluate on a scale whether they would be able to solve it with ease or not? The solution could even be made available, hidden under a button or so, so they can do a self-test without looking at the solution, and checking by looking at it.

I also agree that this would be of interest to the wider Carpentries’ community, and maybe some places have already developed something. Annika, how about posing this question to the Carpentries Discuss list or Slack channel?

By the way, for my bioinformatics master/Phd course we had such a self-test for the unix shell, but the server is no longer operational, the developer moved on and I can’t find the page source anymore...

[drmowinckels]

I was not really thinking about coding exercises for this. But rather questions regarding where people are at so they could be guided. I also think its not a good idea to give coding excersises for this, as it might deter the beginners a lot, making them think they need to know something before coming.

Like the tidyverse course is good for people knowing R, but not tidyverse, as well as beginners. Asking those stating they have experience about certain functions for instance, can give us an idea of where they are at.

If a learner knows and understands the R apply functions then the SWC R is likely too basic, but if they have never done tidyverse its still relevant (particularly the second day).

This also goes for git, where I think basic bash understanding is necessary to really get anything out of it. If a learner has no experience with what cd cpetc is, then we can give advice that they shoudl attend a bash course first, then come back for git after that.

All this can be simple questions, without needing to code. Asking them to make evaluations. Or even, asking them coding questions without asking them to code per se:

What does cd do in the command line?

1. move a file
2. copy a file
3. alter the directory you are in
4. look into a directory

for example.

[annefou]

I have no strong opinion on whether we should or shouldn't have self-assessment; just that is is not easy to set-up and maintain. Another approach could be (as we do now) have two workshops on the same subjects with two levels.

• python/R/git for novices and python/R/git intermediate

We have already done it and it works quite well, especially if the two courses are announced at the same time and objectives and levels are clearly set.

[karinlag]

Hi all!

Lex, correct me if I'm wrong, but I believe the Carpentries central has done quite a bit on assessment, and what works/doesn't work - maybe there's something there we can use?

Other than that, I also like the idea of more advanced workshops, but again, it would be a challenge to ensure that the attendees are at a stage where they would be a good fit for the workshop.

[Teebusch]

I took a bunch of courses with Flames in Belgium. They always used simple but effective self-evaluation in the course descriptions. Here's an example from a graphics in R course.

PREREQUISITES Participants should be able to read in and manipulate data in R. If you have no idea what the following commands mean, the course is too advanced for you:

mydata <- read.table (“c:/temp/rawdata.txt”,header=T, dec=”,”)
sub.males<-mydata[mydata$sex==”male”,]
sub.females<-mydata[mydata$sex==”female”,]
mydata$pass<-as.factor(ifelse(mydata$examresult<10,0,1))
levels(mydata$pass)<-c(“fail”,”pass”)
table(mydata$pass)

(automatically graded exams on repl.it don't seem like a good option. They have only limited support for R. One can't install packages easily. Too much of a hassle, I think.)

(Sidenote: As usual, R has a package for everything, including one for making exams: http://www.r-exams.org/)