search

mllg / batchtools

Tools for computation on batch systems
https://mllg.github.io/batchtools/
GNU Lesser General Public License v3.0
171 stars 51 forks source link

Docker images to accompany example template files #148

Open wlandau-lilly opened 6 years ago

wlandau-lilly commented 6 years ago

@mllg As you say in wlandau-lilly/drake#113, there is no *.tmpl file that works on all installations of a given job scheduler. So as well as the template files themselves, I think we should include Docker image files for which they work.

HenrikBengtsson commented 6 years ago

Gathering @mllg's findings from https://github.com/wlandau-lilly/drake/issues/113#issuecomment-339598735:

Do you happen to have any recommendations of simple/stable images on Docker Hub? I used these images to test the ins and outs of some schedulers:

agaveapi/slurm agaveapi/gridengine vimalkvn/docker-openlava-lsa

I did not find any images with R installations though.

HenrikBengtsson commented 6 years ago

Having a set of Linux containers (LXC) - and let's focus on Docker containers for simplicity - for testing R batch processing on various HPC schedulers would be of great value.

I think the natural common denominator, where such an effort could/should originate from, is the batchtools package (here) rather than in downstream packages (e.g. future.batchtools, drake, ...). It could be that eventually such efforts warrant a separate project, but for now I think it fits well in the batchtools project.

I don't know Docker well enough, but if one easily could create new containers by inheriting from two parent containers that would be awesome, e.g. rocker/r-base + agaveapi/slurm. However, I doubt that can be automated - I'd imagine one need to decide to automate with one parent and then cut'n'paste configs from the other. So, the question is, is it easier to inherits from rocker/r-base (or some derivative of that) and then add installation/setup/config from the other HPC-scheduler containers?

wlandau-lilly commented 6 years ago

Batchtools would be an excellent landing place. Alternatively, do we also want to think about a batchtools-specific repo at https://github.com/rocker-org? @eddelbuettel and @cboettig, would you be open to something like this?