[WIP] Feature/slurm - Githubissues

riga / law

Build large-scale task workflows: luigi + job submission + remote targets + environment sandboxing using Docker/Singularity

http://law.readthedocs.io

BSD 3-Clause "New" or "Revised" License

98 stars 41 forks source link

[WIP] Feature/slurm #95

Closed bfonta closed 2 years ago

bfonta commented 4 years ago

Todo's:

[x] JobManager: add job id retrieval in submit()
[x] JobManager: implement cancel()
[x] JobManager: implement query()
[ ] JobFileFactory: add input/output file handling
[x] Add workflow.

Closes #37.

riga commented 2 years ago

@b-fontana It's been ages since we discussed this the last time, but I finally got access to a slurm cluster and implemented the workflow support in https://github.com/riga/law/tree/feature/slurm_workflow.

Feel free to give it a spin on your cluster if you find the time :)

bfonta commented 2 years ago

In the meantime I have unfortunately lost access to the SLURM cluster that triggered this feature request, but I would be happy to help if your machine can be also accessed by me for a short while.

riga commented 2 years ago

Ah, that's unfortunate, but I somehow already expected this ... it indeed has been a while. I don't think that limited access is possible at the cluster that I'm using, except your already have / had a DESY account.

@tolange In case you do some tests, feel free to log your findings in this PR for future reference :)

riga commented 2 years ago

Slurm workflows are finally there 🎉 → https://github.com/riga/law/releases/tag/v0.1.7

Thanks to @tolange and @kandrosov for testing!