One of the ideas of the option-based configuration of clustermq is that code should be portable between compute environments, e.g. when moving to a different HPC scheduler.
If moving from HPC to local parallelization (even on a heavy duty server), resources will be much more limiting.
For a multicore setup, it would be nice to provide options for setting limits on total concurrent jobs (defaulting to cores/available cores) and memory (defaulting to memory available on that machine)
This way HPC scripts would not need to be adjusted, but could run (at reduced capacity) on multicore environments
One of the ideas of the option-based configuration of
clustermqis that code should be portable between compute environments, e.g. when moving to a different HPC scheduler.If moving from HPC to local parallelization (even on a heavy duty server), resources will be much more limiting.
For a multicore setup, it would be nice to provide options for setting limits on total concurrent jobs (defaulting to cores/available cores) and memory (defaulting to memory available on that machine)
This way HPC scripts would not need to be adjusted, but could run (at reduced capacity) on multicore environments