hpc-carpentry vision

[psteinb]

in many debates around hpc-carpentry, people start to ask for more advanced material on MPI library Z or parallelisation paradigm/library Y or accelerator X. I ackowledge that this "featuritis" is inert to the HPC community (no judgement implied by this).

I propose to formalize our vision more concretely. This will help newcomers directly grasp where hpc-carpentry wants to go and where it will not go, e.g. that we focus on teaching concepts rather than technical details.

Moreover, I suggest to explicitly emphasize that hpc-carpentry limits itself to an onboarding material. This would make it more inline with the other carpentry activities.

Thoughts and feedback is highly welcome!

[ocaisa]

I can see where you are coming from, particularly since it gives a clear immediate goal: define what is implied/required by onboarding and create the lessons to support that.

However, I can also see how this can be somewhat demotivating since it will mean HPC Carpentry will never cover MPI or OpenMP or CUDA or Julia or SyCL or... I would be more inclined to support this if I knew we were willing to create an HPC Carpentries Lab to collect (and in some way support) lessons that go beyond this set (and we already know of a few that do).

Another reason I put it like that is that if we ever apply for funding to support HPC Carpentry, saying that all we will do is to teach people how to log in and submit a job (I know this is cynical interpretation) is unlikely to sit well with reviewers whose expectations are that we reach for the stars.

[psteinb]

Yes, indeed we have to decide where onboarding ends and where it starts. In my view, I am happy to let onboarding end after one or two concepts of OpenMPI/CUDA/Sycl/... have been contained.

[tkphd]

Yes, clearly defining the scope of this Carpentry – where it stands, and where it's going – is important. I'm most familiar with Software Carpentry (git, shell, python). It's hard to put my finger on the line they draw between novice and intermediate concepts; they definitely don't touch anything advanced. I tend to agree with @ocaisa that we need something more than "log in and run this simple task." Teaching the basics of an OpenMP and an MPI program would be great, in my opinion -- I think these are "experienced novice" territory.

I think teaching how to write CUDA or Kokkos code would be a great, but these feel akin to branching (for git) or Python/C API, probably beyond the scope of "introductory" material. If we can find suitable existing tutorials, pointing to those would be great. Otherwise, we'll have to plan to write & maintain such intermediate material as our Carpentry grows, and have some hard conversations about what lessons get archived along the way.

[psteinb]

This is an interesting discussion. We need to bring that to the broader community for sure.

Given the feedback from the carpentrycon@home session, concentrating on teaching the concepts of HPC would my main goal for HPC Carpentry. And yes, this is solved for by submitting simple jobs and understanding their outputs.

As you've seen, hpc-in-a-day https://psteinb.github.io/hpc-in-a-day covers:

• shared memory parallelisation https://psteinb.github.io/hpc-in-a-day/02-02-parallel-estimate/
• distributed memory parallelisation https://psteinb.github.io/hpc-in-a-day/bo-01-bonus-mpi-for-pi/
• high-throughput computing https://psteinb.github.io/hpc-in-a-day/02-04-mapreduce-of-files/
• map-reduce style parallelisation https://psteinb.github.io/hpc-in-a-day/02-03-a-paradigm-shift/

Leaving the discussion aside, if the estimate-pi example is the best ever, I wonder if the concepts conveyed in these lessons is enough for onboarding HPC users. I believe they are and would be the basis for someone, e.g. starting to work on developing GPU code.

𝘌𝘥𝘪𝘵𝘦𝘥 𝘵𝘰 𝘳𝘦𝘮𝘰𝘷𝘦 𝘔𝘢𝘳𝘬𝘥𝘰𝘸𝘯, 𝘸𝘩𝘪𝘤𝘩 𝘪𝘴 𝘭𝘰𝘴𝘵 𝘪𝘯 𝘦𝘮𝘢𝘪𝘭𝘦𝘥 𝘤𝘰𝘮𝘮𝘦𝘯𝘵𝘴.

[HaoZeke]

A quick look through the material (except for the Introduction course) seems like the focus is on parallel programming. Shouldn't we instead focus on:

• Data provenance --> AiiDA and Snakemake
• How to install new software (without root) --> starting from brew to manual installs to Lmod
• How to ensure homogenous environments across the cluster --> pkg-config paths and more
• How to effectively interact with the data stored (given the GUI handicap) --> scp, TRAMP, ssh and more
• Profiling on the HPC (not with time)
• Parsing logfiles
• Hardware utilization and optimizations

[psteinb]

Hey @HaoZeke thanks for chiming in. Those are wonderful ideas. Thanks for sharing.

But keep in mind a couple of things and potentially re-evaluate that list:

• mind our learner profiles, points like How to ensure homogenous environments across the cluster are in my view very advanced for this crown
• we want to target the majority of learners that fall in our learner profile, so Hardware utilization and optimizations, Parsing logfiles (and I admit I am not too sure what you mean by this), How to install new software ... all of these 3 points fall into a very advanced curriculum given the learner profile.

Let's start by shaping a good training material for the majority of novice HPC users, establish a working and fair community around that, then we should approach the advanced topics that many in HPC jump to right from the start.

[reid-a]

This has obvious relationships to Issue #54 about audience selection. Now that we have regular coordination meetings, this issue is maybe less important, but the content identified should not be lost.

[reid-a]

See also Issue #63 for maybe capturing some of the links.