search

calliope-project / euro-calliope

A workflow to build models of the European electricity system for Calliope.
https://euro-calliope.readthedocs.io
MIT License
31 stars 18 forks source link

Improve the cluster interface #93

Open timtroendle opened 3 years ago

timtroendle commented 3 years ago

Right now, Snakemake schedules jobs on the cluster by running a LSF scheduling command. When the job is almost finished, it writes a flag-file to disk, which tells Snakemake the jobs has finished.

This approach is brittle because it cannot detect cases in which the cluster kills the job due to exceeded memory of run time limits. Snakemake waits forever for these jobs.

timtroendle commented 3 years ago

Two possible solutions:

timtroendle commented 2 years ago

Modern Snakemake has cluster-status and cluster-cancel which should be helpful here.

timtroendle commented 2 years ago

This may be superseded by #244.