Personal HTC pool for local testing

[mtwest2718]

I would like to run an hour crash course on HTCondor utilizing the new functionality of a Personal Pool that you can run on a single machine like a laptop with minimal resource overhead. The material will be leveraging the tutorials designed by the CHTC group from the University of Wisconsin, Madison.

• Personas
  • researcher unsure about scaling their work on a cluster
  • system administrators looking to test out a new batch WfMS
• Prerequisites
  • some python and bash experience
  • preferably have conda installed, though this isn't necessary
  • some familiarity with a batch system on a computing cluster
• Objectives
  • Install htcondor software, preferably from conda
  • Start up a Personal HTCondor Pool on their own machine
  • Use a variety of Bash tools to inspect resource usage while the pool is running
  • Query the status of the pool
  • Submit, Hold and remove jobs from the queue
  • Writing a simple DAG workflow

[bast]

What I would add to the proposal which may also help deciding/helping: can you elaborate on typical use cases where HTC can help a researcher or sysadmin?

[mtwest2718]

• Why an HTC design would be preferable to HPC solution in many cases.
  • Many researchers come to an RSE with a pleasantly parallel workflow which do not high-speed node interconnects or super-fast file-servers. They are working with narrow applications that have been developed on a desktop with single-thread performance in mind and a cluster provides a means to scale these workloads (like a Monte Carlo).
  • An HTC design limits the amount of change the researchers code needs to make and just asks them to learn the batch system. There are no OpenMP or MPI modules to learn and integrate into the code. What design changes that could be made make the overall workflow more robust and reproducible.
  • My flavor of batch system, HTCondor has an included meta-scheduler that can relate different types of job submissions together with a dependency graph. There is also a Python API for constructing submit & DAG files along with job submission so one can avoid working in Bash (in most instances).
  • Because you don't care as strongly about system requirements, distributed computing across multiple clusters is far easier.
• And on the question of why a personal pool might be helpful
  • Using a DAG allows job submissions to be done automatically in a predefined manner. So preprocessing, analysis, results collection and plotting can all be handled in one big organized batch rather than distinct batch submissions.
  • Researchers can be unsure about structuring their work for a cluster and hesitant to do active development on the main system. Providing a means to design and test their submit files on their own machine would give them more confidence that when they do move it over to a cluster, it would work the first time.
  • The personal pool allows admins to test out different configurations on an isolated & simple system before porting it to a larger scale cluster or multi-machine pool.

[mtwest2718]

Some workloads do need HPC methods to work efficiently and that is fine.

[mtwest2718]

I do like this presentation for how it lays out the differences between HPC and HTC perspectives.

[mtwest2718]

Oh and it's cool that Richard and Simo found a Docker container set to setup a SLURM cluster on a single machine. So local pool for testing isn't an unreasonable idea.

[bast]

Just confirming also here that we are looking forward to have this in the program and we will together fine-tune the timings (we are still unsure about how much time we have for sessions since we are waiting for feedback from others but we are working on it and will agree on this together with you).

[bast]

This is now in the program: https://nordic-rse.org/events/2020-online-get-together/#tuesday-december-1st Can you please suggest an abstract/outline? I guess we can reuse many of the good points above.

[rantahar]

Closing, this was proposed for the 2020 workshop