Prevent simultaneous pulls of Singularity images by toil-cwl-runner: add locking

DataBiosphere / toil

A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

http://toil.ucsc-cgl.org/.

Apache License 2.0

894 stars 241 forks source link

Prevent simultaneous pulls of Singularity images by toil-cwl-runner: add locking #4990

Closed unito-bot closed 2 months ago

unito-bot commented 3 months ago

We should make sure that, even if there isn’t a shared filesystem for a cluster, we don’t have several CWL jobs on the same node trying to simultaneously pull the same Singularity image, because this can corrupt Singularity’s internal files.

This is in addition to any prefetching we might want to do when we do have a shared filesystem.

┆Issue is synchronized with this Jira Story ┆Issue Number: TOIL-1603

unito-bot commented 2 months ago

➤ Adam Novak commented:

Steven says adding locking around the actual pull in cwltool would require some monkey-patching. Maybe we can pre-pull?

unito-bot commented 2 months ago

➤ Adam Novak commented:

Moving some CWLtool code that determines the container type to use might make this easier to implement.