notes: SLURM CI setup used in ReproMan

[yarikoptic]

Might be of interest since you have mentioned desire to setup CI for various runners. Unfortunately during our chat I even forgot we did have basic testing of slurm setup. So I have traced back all the components and decided to outline for myself, @asmacdo and everyone who might be interested ;)

Here is what we have in reproman for slurm:

• https://github.com/ReproNim/reproman-slurm has dockerfile from which we built repronim/reproman-slurm which we have on docker hub: https://hub.docker.com/r/repronim/reproman-slurm
  • seems listed as sponsored OSS project so should stay there and may be even rebuild automagically if we change Dockerfile?
• https://github.com/ReproNim/reproman/blob/master/tools/ci/setup-slurm-container.sh is the script we use in our github action for testing: https://github.com/ReproNim/reproman/blob/HEAD/.github/workflows/test.yml#L102
• we use pytest for testing, so we designed no_slurm check which is used by code below to come up with skipif.no_slurm() helper so we skip test if there is no slurm available in that test matrix/environment
• then we came up with ssh_slurm fixture to be used in tests: https://github.com/ReproNim/reproman/blob/HEAD/reproman/support/jobs/tests/test_orchestrators.py#L68 and it is used by 2 tests: test_orc_datalad_slurm and test_orc_datalad_concurrent_slurm.

hopefully this comes useful to any degree. Feel free to clone upon reading ;)

on that note -- also briefly looked into what testing psi-j does. https://github.com/ExaWorks/psij-python/blob/main/README-testing.md describes an interesting setup presumably (haven't spotted some dashboard yet) allowing different clients to submit results back to them. We have similar-ish basic setup for testing of git-annex (and potentially datalad) on various client systems in https://github.com/datalad/git-annex/#client-tests (defined in https://github.com/datalad/git-annex/blob/master/clients/clients.yaml). So - if interested to test git-annex and/or datalad on your system and contribute back to our dashboard -- let us know ;) Details are available in https://github.com/datalad/git-annex/tree/master/clients#testing-git-annex-builds-on-local-clients

[zhao-cy]

Oh wow, these are really helpful and informative! Really apprepriate it, Yarik!

[zhao-cy]

I just wanted to add another potential Slurm docker image resource here which mentioned by Dorota today. However it was released in Aug 2019, which is older than what Yarik suggested.

I'll probably try out what Yarik suggested first!

[asmacdo]

Link dump:

PSI/J: https://github.com/ExaWorks/psij-python Gc3pie: https://github.com/gc3pie/gc3pie Babs: https://github.com/PennLINC/babs Pegasus: https://github.com/pegasus-isi/pegasus Nextflow: https://github.com/nextflow-io/nextflow Snakemake: https://github.com/snakemake/snakemake