hpc / Spindle

Scalable dynamic library and python loading in HPC environments
Other
94 stars 22 forks source link

Adding development / tutorial container #43

Open vsoch opened 3 years ago

vsoch commented 3 years ago

This is the start of work to add a small development environment to serve two cases:

  1. a quick and easy example for a new user to run to install and use spindle
  2. a development environment

The environment include a two node slurm cluster with centos nodes, and of course the main "login" node that the user can shell into to interact with the job manager and also install / use spindle. It is done by way of docker-compose and a new folder "docker."

A container build could also be useful to build and test spindle in CI, although that is not included here. To finish up this work I need a little help with a dummy example that can quickly show the advantages of using spindle. I had wanted to do something with Python loading (e.g., having a library that sleeps 5 seconds and showing the load being 5 vs 10 seconds) but it's not clear to me how to use spindle (the only examples are for using mpi). Is this a reasonable use case or should I install mpi and try to reproduce that example?

Finally, my last question is about the scope of user base for spindle. Is this a niche application that is only interesting to a small subset of MPI users/developers with a very specific problem, or is there something here that we could share with the larger community and say "this solves a problem that you have, here are a bunch of examples for why /how to use it."