Discussions about HPC Carpentry curriculum

[tracykteal]

There has been broad interest in the collaborative development or discussion of training for using high-throughput or high-performance computing. A call initiated by Greg Monaco at Great Plains Network led to a discussion with a group of interested people.

This issue is to continue that discussion and make people aware of ongoing efforts in the development of 'HPC Carpentry' curriculum.

This thread is the outcome of a meeting with notes here http://pad.software-carpentry.org/hpc-carpentry

[gvwilson]

We tried to get a lesson on HPC going in Software Carpentry as part of the ACI-REF instructor training, but it never got off the ground. What little we have is at https://github.com/swcarpentry/hpc-novice.

[apawlik]

Some time ago I created an organisation https://github.com/HPCCarpentry Thanks to @callaghanmt from the University of Leeds we now have https://github.com/HPCCarpentry/organisation/blob/master/overview.md

I also bought a domain (hpc-carpentry.org) and trying to find time to adopt the template https://github.com/HPCCarpentry/HPCCarpentry.github.io

There is a number of people interested in this so maybe it'd pick up. @tracykteal would you be able to put me in touch with Greg Monaco at Great Plains Network ?

[dbrunson]

Given the previous attempts at making HPC Carpentry lessons, do you have any advice on how to keep the momentum going? What do you think kept the previous attempts from getting off the ground?

[gvwilson]

In my experience, a lesson needs to start with a complete (if rough) first draft that others can critique and extend. In this case, I'd suggest:

• Create half a dozen Markdown files, one for each topic in the lesson, each containing a dozen bullet points outlining what that half-hour topic will cover.
• Include one fully-worked-out exercise in each Markdown file to show what learners will be able to do at the end of it. (I think this needs to be fully worked out, rather than a vague hint like "something with job submission").
• Ask for PRs and issues adding/removing/rearranging/critiquing bullet points.

We're doing something like this with https://github.com/swcarpentry/good-enough-practices-in-scientific-computing (a paper rather than a lesson, but the principle's the same), and it's working well: there only needed to be this much for people to start leaving comments, at which point it gathered speed very quickly.

[ChristinaLK]

V. much agree with @gvwilson - someone needs to do the first pass of outlining/writing and then there's something that everyone can discuss/modify.

[tracykteal]

Yes, we've found that starting with 'what to teach' is a good way to move forward. Then we can start to draw from existing resources to put the lesson together. We're working on that 'what to teach' document, and this is a start

https://github.com/datacarpentry/hpc-carpentry/blob/gh-pages/what-to-teach.md

We're setting up a call to do more work on this document. If you're interested in participating, please let me know or note it on this issue. It would be great to have more perspectives!

[ghost]

HPC@LSU has a series of Moodle tutorials at http://moodle.hpc.lsu.edu that use a Creative Commons Attribution non-Commercial license. Anyone with a Google account can use them. And the source can be provided for local customization. I tried to make a Wiki entry, but was rebuffed.

[hneeman]

I know I've become a bit of a bore about this, but:

I encourage those of you who aren't familiar with "Supercomputing in Plain English" workshops to take a look:

http://www.oscer.ou.edu/

I would imagine that (portions of) these materials would have value in the context of HPC Carpentry.

We've taught over 2000 people from 362 academic, government, industry and nonprofit institutions and organizations in 51 US states and territories and 17 other countries. We provide this series every other spring (next one is expected to be spring 2017), live via videoconferencing, for free.

We also have little exercises that introduce the basics of Linux, batch computing and running parallel jobs on a cluster, which are well tested (i.e., we've used them many times and gotten good informal feedback -- I don't want to pretend we've done formal research on their effectiveness) and have been used by a number of courses for HPC novices.

We've also used (portions of) these materials at workshops taught and/or hosted by the National Computational Science Institute (http://www.computationalscience.org), the Linux Clusters Institute (http://www.linuxclustersinstitute.org), the SC supercomputing conferences, the Rocky Mountain Advanced Computing Consortium's HPC Symposium (https://www.rmacc.org/hpcsymposium), and other venues.

[vuw-ecs-kevin]

I note that the 'what to teach' document says

"We've started a document to discuss these ideas focusing on an user who has little to no prior computational experience."

and I'd like to suggest that trying to teach HPC to users who have "little to no prior computational experience" will not be good for them, nor for you, the teacher !

Reading the document that the what-to-teach.md document links to though makes it clear that much of what needs to be taught to people already using a computer to do research but who now need to use "beyond the desktop" computational resources, is some insight into what it means to leave their PC (with the emphasis on the P for Personal) and go off and use remote, but more importantly, potentially shared with other users, computational resources.

Software Carpentry already has a "shell extras" lesson (which may have fallen by the wayside ?) that introduces the user of a "personal computer" to the joys of remote computing, so you probably want to look at teaching people what they'll find once they have SSH-ed into a remote machine that either offers "beyond the desktop" resources or is the front-end machine for such resources.

Here are some questions that keep popping up and to which, I'd suggest, the answers to might serve as a basis for some of the HPC Carpentry episodes.

1) I have hard-coded my username and filenames into my code and they won't let me have me the same username nor directory structure on this remote machine that I have been told to use - what should I do ?

2) I managed to run 100 instances of my code but all I got back was a single file called output.txt - where are the other 99 files ?

3) No, I don't have small version of the problem that I can use for a dry-run on my own machine to make sure that things are likely to work as I hope they will now I have access to all these shared HPC resources - why would I spend any time doing something that wouldn't produce results that I could show to my supervisor ?

4) My program was developed in, and is run from, the Matlab GUI and so all the variables are splattered throughtout the code because it is one monolithic file that I've been editing every day since being giben it to work with - how am I goingto run 100 copies of this program in parallel without having to edit 100 copies of the M-file by clicking on the Matlab icon (and where are the icons anyway - all I've seen so far is a dollar sign?) 100 times to fire up 100 instances of the GUI to do so?

Given that most HPC computing will require the use of remote resources, for various values of "remote" before starting to teach HPC, look to teach people, who are clearly no longer being taught to do so, to write PORTABLE code that can easily be transPORTED from their PC, the one they will be sitting at every day and so will have become very used to, to another machine: you may well find that all that's needed is the getting across of those same, simple, basic

input -> program -> output

concepts (UNIX pipeline philosphy anyone?) that used to be taught before GUIs came along and made it easier for people with "little to no prior computational experience" to use computers without actually gaining "much, if any, computational experience" whilst doing so.

To put that another way, before looking to run any HPC Carpentry lessons, maybe we should be knocking up some (Non-Personal Computing) NPC Carpentry ones?

The logo for such an effort could be the other end of that claw hammer from the Software Carpentry banner, being used to extract a nail, as a methaphor for the undoing of the structures that PC users might need to "unlearn", in order to do HPC?

Tounge-in-cheekness and slight cynicisms aside, some of the concepts that HPC Carpentry needs to promulgate also have the potential to make our researchers better users of their non-HPC resources, so perhaps some of what's being flagged up as part of your HPC syllabus shouldn't really be being delivered under an HPC banner, because that might stop some people, who would benefit from being exposed to the content, from discovering it.

[vuw-ecs-kevin]

Over the weekend, I was thinking about where I "came in" to this arena and, more importantly, why now, I can often see what it is that some of the people I'm supporting are not quite getting, and so, further to my previous posting, which argued that a lot of the knowledge required to "do HPC", needn't necessarily be taught as if it were inherently "HPC", I'd like make a further suggestion, around the idea of HPC Carpentry being perhaps, the wrong name, by putting forwards the idea of thinking baout what to teach as being more of an "HPC Scaffolding", as follows.

Whilst HPC needn't necessarily be parallel, one way to look to get some of the underlying concepts across might be to hark back into the history, and the days of the transputer, and OSes like Trollius and LAM (Local Area Multicomputer), which had the notions of "Inside The Box" and "Outside The Box" computing, along with, in certain cases, a need for the user to think about exactly where, within the "processor" toplogy (NaIL files anyone?), they would need to explcitly place their "processes", plus the need to multiplex data streams across that topology.

Some exposure to that way of thinking - about process, and indeed, data, placement, clearly has parallels (no pun intended) into today's realm of HPC where, although the placement may be somewhat hidden from the user, by a package or a toolkit, it's still happening, via in-program hand-offs to thread-pools or GPU cards, or via external local resource manager (LRM) scheduling of programs themselves, across compute resources - would then serve as more of a "scaffolding", for users who already have the basic "Software/Data Carpentry" tools, on which to scale up (pun intended) and so be able to do work on the "HPC tower".

Hoping this line of thinking might be useful, whilst being aware that it clearly shows my age.