General goals for shell lesson

[k8hertweck]

In a recent thread on the discuss list about the use of particular text editors for this lesson, the discussion moved into a more general conversation about the goals for teaching shell and how the lesson fits with the needs of other lessons taught at the same workshop. The SWC leadership would be interested to hear how our community members feel about the utility and learning objectives of the shell lesson and your experiences teaching it.

[ctb]

I've taught various versions of shell workshops for over a decade. In my field (genomics and bioinformatics, specifically) and with the methods that we use for getting things done (single-computer and queue-based HPC) on a variety of computers (laptops, cloud, docker, and HPC), I value:

• one editor that can be used everywhere in the shell (local and remote);
• methods that can serve multiple purposes (shell scripting for automation, reproducibility, and qsub scripts);
• methods that have a long history of working for many purposes and for which there are many non-SC/DC tutorials (shell!)
• language/toolkit methods not tied to specific higher-level computing (sh/bash as independent of R and Python)

[justbennet]

Posted to the wrong place. Apologies, and repeat here, for the record.

Since I may have started some of this, please let me clarify my original concern. It was specifically these two paragraphs.

I think that in the past, one learned shell to move around, nano to edit, then one used nano and the shell to create R or Python files, which were run from the command line, and all of the code was under Git control which was also run from the command line. That scheme has an internal coherence, it builds skill upon skill, and there is a recognizable goal.

The lessons seem to be getting more and more independent and less and less like parts of something bigger -- at least that's how I would see it from a participants perspective and based on the anecdotes I have from people who've been to recent camps.

I think the overall idea of a command-line, scripted workflow is getting lost in the local concerns within lessons which are more and more GUI-oriented and independent of the command line. If none of the other lessons use the shell, then what is the point of the shell lesson?

One of the things that first drew me to SWC is that it appeared to have a coherent, end-to-end philosophy, and the lessons supported that and made that philosophy more clear. That has diminished, and the connections between the shell lessons and the others are now more tenuous.

As I said in the original post, maybe the world moves on, and I will deal with that on my own.

I just thought I would try one more time to make clear that it is the relationship among the lessons that I feel is not getting enough attention. I thought the original SWC scheme was well-connected from lesson to lesson. I think that is less so now.

My due diligence is to try to help make that a conscious and explicit decision and not the result of drift. If that is where people want to go, all the best!

Sorry for not being better at communicating.

[shwina]

I think that the following is dangerous thinking:

1. The shell lesson is about using a command-line.
2. Many (most?) people prefer using a GUI for programming or version control.
3. Therefore we should drop the shell lesson.

Software Carpentry is about learning "lab skills in research computing". We don't teach learners detailed description and usage of any specific lab instrument, but rather more general (and in my opinion more important) concepts, such as "clean instruments regularly", or "keep a lab notebook".

We don't teach Python or R, but we do use them as vehicles for teaching important concepts:

1. Write programs for humans, not computers, i.e., prefer readability over performance (in many cases)
2. Don't repeat yourself, i.e., write and re-use functions for common tasks
3. Test code using code

Similarly, we use the shell as a vehicle in this lesson for teaching things like the following:

1. The windows, menus and icons on your computer are not the only way you can interact with it
2. It is better to have/build many small, general purpose tools than fewer large, specific tools
3. Use scripts (which are plain text files) to reproduce results

In my opinion, whether learners will in practice use the shell to run Python scripts, run Git commands, or edit text files is hardly relevant: they still walk away from the shell lesson with important ideas, and the ability to connect them with ideas in other lessons.

[ChristinaLK]

I agree w/ @shwina and would add that shell principles that I find important are:

• the structure of a filesystem and the importance of understanding how your computer is organized
• an introduction to the concept of for loops

Also, to expand point 2 -- introducing the idea of small "functions" which you can then combine for greater functionality.

(also, it's hard to convey to novice learners, but whenever it makes sense (for a more advanced audience), the shell lesson can plant the idea that typing is faster than clicking, so ultimately, command line interfaces can accelerate your work. )

[k8hertweck]

Thanks to all for your comments here so far.

I must admit that I consistently have at least 1-2 students in each workshop I teach asking after the shell lesson "But when would I use this?" Given shell is often the first unit taught at a workshop, this is certainly something I need to work on, and am keenly interested in developing better strategies to motivate student learning (so thanks to those of you who have added ideas here; perhaps we can think about modifying the intro language to the shell lesson to reflect this?).

@justbennet, thanks for getting this conversation going. Since shell often sets the stage for the rest of the workshop, I consider it intricately linked with the overall flow of the subsequent materials. The concern you raise about unity between lessons is valid. I personally think there is a tradeoff between modularity (ability to select particular lessons for a given workshop) and overall flow. Regardless, I think we could do a better job minimizing that tradeoff! FWIW, this is one of the core differences between Software Carpentry's materials and lessons developed by Data Carpentry. The latter are designed as an entire 2-day workshop, and focus on a consistent dataset (or at least type of data) throughout. As others allude, that particular design decision goes back to what specific skills/concepts we're trying to impart, and how we motivate learners. I've been conversing with a few other folks in leadership of the Carpentries and we're thinking about how we'll manage these differences as our organization continues to grow.

[gdevenyi]

@shwina has covered my feelings very well.

I'd like to cover one more thing not yet mentioned.

The terminal can be a nearly uniform interface on windows/OSX/linux/unix. With the addition of ssh, it becomes a remote work tool. This is something that is completely unique, a supercomputer looks the same as your laptop once once you grasp the general principles.

Having said all this, I think the shell-novice may be now trying to sell the shell a bit to hard at doing "real work" with the various builtins. Perhaps we should undertake a careful re-think of the focus of the shell materials as a method to drive other tools, and manage files, rather than doing work with the shell directly.

[justbennet]

Do the current shell lessons lean too heavily on the techniques within the shell, e.g., mkdir, rmdir, touch, and not perhaps enough on demonstrating what can be done with the 'tools' available? I do understand that some people seem to have a hard time grasping verbal explanations of directories in class, but I think it might be worth trying to come up with some examples that resemble some real work flow -- perhaps as a script or set of scripts that do some work -- demontrate the real work flow, and then use that as a framework for going through some or all of the specific details of which shell commands and Linux supplemental programs are used to accomplish them. No, we would not be able to get through all of it, but the participants would be left with examples that they could then explore on their own, try to use as templates, etc. The comments I got from the last shell workshop were basically that no realistic use-cases were presented, way too much time was spent on folders, that the specific commands that were covered were all well and good, and necessary, but they were not put together in a way that made them seem useful at a practical, day-to-day level.

In the terms of this discussion, the concepts were lost on the participants in the details of the shell commands. The concepts were never illustrated, demonstrated, nor explained sufficiently that the people to whom I spoke (about 7 of 20 or so) the workshop realized there even were really concepts or principles that were being elucidated.

Is the forest being lost for the trees? Should we expect and accept more missing of details in favor of showing more capabilities? Can the lessons be modified to better illustrate the core concepts that people want to impart and spend less time on some of the mechanics? Would that be a good thing to try?

Do people think that the material is already there, and perhaps my experience was just one where the execution was off? If the latter was the case, would better material for instructors that explained how, specifically, to explain the core concepts and which concepts related to which material in the Lesson, help insure that execution is more uniform and hits the key concepts?

[shwina]

The comments I got from the last shell workshop were basically that no realistic use-cases were presented, way too much time was spent on folders, that the specific commands that were covered were all well and good, and necessary, but they were not put together in a way that made them seem useful at a practical, day-to-day level.

I am interested to know if this is the general feedback for other SWC instructors as well. Certainly this is not true for the audience I regularly teach, for whom the shell session is almost always very well received. Granted, this audience is beginner users of our HPC cluster: they are especially motivated to learn about the shell..

Do people think that the material is already there, and perhaps my experience was just one where the execution was off? If the latter was the case, would better material for instructors that explained how, specifically, to explain the core concepts and which concepts related to which material in the Lesson, help insure that execution is more uniform and hits the key concepts?

Certainly I wouldn't jump to the conclusion that your execution may have been off. My experience with teaching the shell lesson is that by closely following the lesson material, you can easily spend the majority of the workshop covering just filesystem navigation and manipulation: ls, pwd, cd, mkdir, cp, rm, mv, relative and absolute paths. Personally, I am learning to do a bit of a "speed run" of this initial material and focus more on pipes, loops and scripts.

Can the lessons be modified to better illustrate the core concepts that people want to impart and spend less time on some of the mechanics

By mechanics, I would assume concepts similar to the following:

• . and ..
• relative v/s absolute paths
• using an editor like nano

Certainly we can do with less of the above, and focus more on the "core concepts" pipes, loops, scripts, etc., but the lesser time that is spent on mechanics, the less comfortable users will be with the shell.

I think we can achieve a good balance between the two in a 3 hour workshop, and I think one way to do it would be to diffuse the above "mechanics" into the rest of the lesson, rather than dedicating the first half of the workshop to them.

[justbennet]

Excellent suggestions, and thanks. I was worried that my observations were idiosyncratic and out of touch with other people's experience. Yes, I think far less time can and should be spent on explaining folder structure explicitly.

By 'concepts', I was referring back to your three points from the comments above.

1. The windows, menus and icons on your computer are not the only way you can interact with it
2. It is better to have/build many small, general purpose tools than fewer large, specific tools
3. Use scripts (which are plain text files) to reproduce results

I think it would be beneficial for us, as instructors, and to the participants, if we had some more realistic examples of a plausible 'end product' -- one or two shell scripts to demonstrate the power and efficacy of shell scripts -- that we then use to show some mechanics, creating folders, using loops, error/return code checking, utilities like man, [e]grep, find, tr, cut, sed, awk, df, du, date. Some selection that is useful and representative.

If we provide some shell script examples, that also means that people can take them home, run them, and try to figure out what the parts that are not explained in the workshop do.

If we are telling people that having and/or building small general purpose tools is good, I think we're kind of obligated to show them how to fit them together into something plausibly useful, and get them started with some examples, and some of the components for building one themselves.

[gdevenyi]

I have to bow out of further conversation here, imminent baby arriving. Sorry!

[ctb]

In response to @justbennet https://github.com/swcarpentry/shell-novice/issues/542#issuecomment-292214889, one idea might be to point at the reproducible research curriculum that Data Carpentry is developing. That (should be? is?) a more integrative approach to the concepts taught in the 2 day workshop.

Of course, the overall problem might just be that we can't teach all that much in 2 days... :) :(

I'll post the link here when I find it :)

[ctb]

Via @butterflyology,

The rendered pages with merged PRs are:

http://www.datacarpentry.org/rr-literate-programming/02-literate-programming/ http://www.datacarpentry.org/rr-literate-programming/03-explore-knitr/ http://www.datacarpentry.org/rr-version-control/03-git-in-rstudio/ http://www.datacarpentry.org/rr-publication/01-publication/

There are a few lessons that are still need a little more work but are ready:

http://www.datacarpentry.org/rr-intro/ http://www.datacarpentry.org/rr-organization1/

So I guess this doesn't address the shell question so much.

I'm still a fan of my Oslo repeatability workshop, which integrates shell to some extent --

https://2016-oslo-repeatability.readthedocs.io/en/latest/

[antonangelo]

One of the most challenging things I've found to teach people is file discipline. Where files and directories are, how to manage them, how to manipulate them. I taught web development (and a bit of 'real' coding :D) for a number of years, and getting the idea that there were separate file systems for prod and dev, and why you might use relative vs. absolute links over was fundamental, but relied on some basics that even some pretty advanced computer users were not aware of. A lot of learners seemed to get a lot out of working on some basics.

I want to make a couple of points, and my apologies for verbosity.

A unix shell is a bit like Latin. Once you have grasped the fundamentals, then all romance languages open up to you. The metaphor stretches nicely, as you don;t have to have a lick of Latin to speak Italian, but you've already learned some Latin by osmosis. By being exposed to a unix shell, even if you never use it again, you will have a better idea of How Files Work, and that has a lot of positive down flow effects, especially for the file discipline problem I mentioned above. Appropriate naming of files and folders seems like such a simple thing, but....

I do have some sympathy for those who suggest that unix shell may not be a transferable skill outside academia, like, say, APA referencing. However I think some of the core ideas that you can teach by using a unix shell are very useful to anyone who goes on to work in any computing environment.

Secondly, I'd like to take up the @ctb repeatability banner. Unix shell scripts that work in a non destructive way are a straightforward way to keep your house keeping transparent. Want to see how data has been 'cleaned'? Let me show you diff....

I'd like to support the content current lessons, as they teach a nice set of skills, and if you give the background of the problem you are trying to solve, I think they are really relevant.

I'm interested in @shwina suggestions in focusing on the scripting elements, as most of that could also be done in python (reinforcing python) rather than another 'language'. Unix shell scripting is pretty idiosyncratic. However, python may not be the focus of the event, so keeping it agnostic may be a better idea.

[deppen8]

This is a great discussion, but since it has been inactive for 4+ years, I am closing as part of old issue cleanup. If there is still appetite to continue the conversation, we can reopen. Any changes will likely need the input of the curriculum committee.