solutions for very long-running jobs

cambiotraining / hpc-intro

Practical course on running jobs on a HPC

https://cambiotraining.github.io/hpc-intro/

Other

14 stars 13 forks source link

Open tavareshugo opened 2 years ago

tavareshugo commented 2 years ago

Could add something to the materials for very long jobs that timeout (e.g. on CSD3).

checkpoint:
- https://dmtcp.sourceforge.io/ (need to see how this works exactly)
- another one: https://criu.org/Main_Page
- this may sometimes break things -- make sure to test before you do a big job with it
use cases of breaking workflows into smaller pieces. For example, with GATK can use the new GenomicsDB format to add samples sequentially to the database.
parallelise your code

tavareshugo commented 1 year ago

This could be part of a separate "advanced" module, together with #18