Closed tkphd closed 1 year ago
For me there are a few assumptions going on in general when delivering this course, one of those that you have knowledge/control over the environment of your users. The amdahl package is released in the way it is because it can work in a number of ways: if you have modules you can expose it that way (including encoding it's mpi4py
dependency), if you don't you can still install it via pip (which also checks/installs the required dependency). Just shipping a tarball has a good chance of not working because there is the assumption that mpi4py
is already available in the users environment.
Ultimately this sounds like something that belongs in a snippet as there a selection of available ways to make the code available which are all perfectly valid:
.py
extension, instead change the command to execute the code)Thanks for the thoughtful feedback @ocaisa. My rationalization for the tarball is that Ep. 15 (transferring files) depends on a tarball, and previously, it was repackaged data from a Python lesson that is unrelated and is never used after that episode. This meant a lot of the transfer commands were boilerplate with no concrete filenames or paths, which is mystifying and seems counterproductive. Overhauling Ep. 15 with an amdahl
tarball feels right, because it makes the concepts of transferring more concrete and gives them purpose.
You are right, of course, that mpi4py
may be unavailable, and having the cluster admin install amdahl
as a module or via pip
is sensible. Can you update the README/docs in the [amdahl repo]() to outline the recommended workflow for installation, and usage after it's installed?
I can revise the tarball in this PR to contain a driver script that will simply import amdahl
and run it, so we can still work through the curl
/wget
, tar
, and scp
/rsync
commands in Ep. 15, then run it in Ep. 16.
In today's Coworking Hours, we had a vigorous debate about how to install amdahl. The best/obvious option is to python3 -m pip install amdahl
, which will pull in mpi4py if it's not already installed. Possible breakage modes include
Can folks with access try to install amdahl on their login nodes, and check whether it successfully runs?
python3 -m pip install amdahl
mpirun -np 4 amdahl
Thanks!
I read the notes of the coworking hour where this was discussed and I still think the easiest is to go with a pip install
approach.
As mentioned in those notes, it is possible to do an offline pip install
if you have the tarball available. You can get the latest version of the tarball with the URL https://github.com/hpc-carpentry/amdahl/tarball/main so there would be no need to maintain a versioned tarball in the lesson.
With a tarball on the system (via curl/wget, plus scp if needed), let's just assume that the $HOME
directory is on a shared FS (usually pretty reasonable assumption). Installation would be:
python -m pip install --user <tarball>
This doesn't resolve any dependency problem you may have, but the most problematic one for our use case is MPI4PY. This can be installed with
env MPICC=/path/to/mpicc python -m pip install --user mpi4py
While I haven't tested it, I imagine that mpi4py
also be replaced by a tarball in this command. The MPICC
environment variable is only really needed if mpicc
is not in your environment (or maybe you need a particular MPI C compiler).
Following this month's meetings, I have a few things to change on this branch before it's ready to merge -- mostly, teaching pip
as the installation method of choice with some caveats for mpi4py
and offline installation instructions.
Using the Amdahl/MPI4PY tarballs to exercise "transferring files," make sure to cover the cluster configuration firewalls allowing users to "pull" from inside, "push" from outside, or both.
I think
hpc-intro-code.tar.gz
is still being shipped with this PR
Weird, I thought 8a6f91b832ce6f4e02c37e0e899343d5b7580667 deleted it. Taken care of in decb121, which also consolidates the lingering pi code in a new tarball.
Maybe tools like https://www.parallelpython.com/ , https://www.dask.org/ and https://github.com/pgiri/dispy should be mentioned? Not all people will need MPI, and these might provide good alternatives when task farming is the main purpose.
Maybe tools like https://www.parallelpython.com/ , https://www.dask.org/ and https://github.com/pgiri/dispy should be mentioned? Not all people will need MPI, and these might provide good alternatives when task farming is the main purpose.
Out of scope for this PR. Please file a separate issue or, even better, a PR setting MPI in context with these other tools/approaches, bearing in mind that there may be controversy over whether task farming is HPC (vs. HTC).
I believe the requested changes have been made. Please re-review.
Many many thanks @tkphd for putting so much effort into this PR!
This PR re-opens #407, which was merged prematurely, and relates to #409. Original summary and quoted comments follow.
This PR replaces the dummy
hpc-intro-data.tar.gz
with a tarballed version of@ocaisa'sour amdahl program,hpc-intro-code.tar.gz
, and substitutes that black-box program in place of the pi calculator. This is in keeping with our goal of reducing the "programming" aspect of hpc-intro to the bare minimum, focusing instead on the cluster and practicing job submission.Note that, whereas the upstream amdahl package can be installed as a module, the version archived here is a stand-alone Python file named
amdahl
, with defaults of 30 seconds of work and a parallel proportion of 85%.@reid-a and I are planning to teach this version on Friday, 24 June (
two days hencetomorrow).@ocaisa wrote,
@tkphd responded,
@ocaisa commented,
@tkphd asked,
Edited to update Amdahl repo URL