carpentries-incubator / hpc-intro

Lesson materials for an Introduction to High Performance Computing in the tradition of Software Carpentry
https://carpentries-incubator.github.io/hpc-intro/
Other
132 stars 129 forks source link

Active maintenance of snippets_library #317

Open ocaisa opened 3 years ago

ocaisa commented 3 years ago

As seen in #296, I really don't think it is reasonable that we ask the contributor to create/edit the snippets for all the different site permutations. I think @mikerenfro went above and beyond to do that, and it will definitely turn people off contributing non-trivial stuff if that is considered necessary. That's not to mention the fact that it is virtually guaranteed that they will not have access to these systems so can't actually verify that the new/edited snippets even work.

Our current reference system is Graham at Compute Canada, do any maintainers even have access to that system?

tkphd commented 3 years ago

If the modified lesson requires a new snippet, then checks will fail if the snippet library is not updated. This does not require the originator of the PR to write all the new snippets, and perhaps they should not, as there will be missing expertise. The failures resulting from such a PR should be brought to the attention of experts -- either the owners of the particular clusters, or people sufficiently versed in the resource manager to make a close approximation.

Another approach would be to template as many things as we can think of, then refactor the snippet library based on the queuing system, so there's only one file -- massive though it may be -- to edit.

ocaisa commented 3 years ago

I'm not a fan of massive amounts of templating, it makes it a little hard to know what to update when you review the lesson: to know the context you have to first look at the rendered lesson, then find the line in the source to know the template being used.

Also resource requests for SLURM are over multiple lines, while for PBS they are colon separated on a single line, that would be really hard to get to work correctly for both cases.

tkphd commented 3 years ago

Agreed, we would have to document the _config.yml and template variables carefully to make clear what values a new site should put in which fields. This is a challenge.

We would maintain separate "libraries" for Slurm vs. PBS/Torque, etc., since there are incompatible syntax differences.

Beyond such a sweeping change, which would bring substantial difficulties of its own, it's unclear that this issue is actionable as written.

bkmgit commented 2 years ago

Generic SLURM and PBS examples are sufficient, one can leave a few of the sites for people who wish to use them for further examples for customization if they need them.