search

USCbiostats / slurmR

slurmR: A Lightweight Wrapper for Slurm
https://uscbiostats.github.io/slurmR/
Other
58 stars 12 forks source link

[JOSS review] Chunking of jobs #10

Open mschubert opened 5 years ago

mschubert commented 5 years ago

This is a part of the JOSS review outlined in https://github.com/openjournals/joss-reviews/issues/1493.

As far as I can tell, individual function calls are always submitted as array jobs.

What if you have millions of calls? Does that mean sluRm will submit an array with millions of entries?

In other words, can sluRm chunk together multiple function calls in one jobs?

I assume it can not. This is fine, but should be documented.

mschubert commented 5 years ago

Looking at e.g. Slurm_lapply, it seems I was wrong and chunking is supported :+1:

Please document!

gvegayon commented 5 years ago

Yes! Of course, if that were not the case, then ppl should go to https://github.com/mschubert/clustermq instead :). Documented.