search

marbl / verkko

265 stars 27 forks source link

How to rerun an interrupted task? #242

Closed KANGYUlab closed 2 months ago

KANGYUlab commented 3 months ago

The task was interrupted due to queue and stopped at 2-processGraph process. Can I continue running from an interrupted process? How? Thanks a lot. The end of the snakemake.log file is as follows:

`rule processGraph: input: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/paths.gaf output: 2-processGraph/unitig-unrolled-hifi-resolved.gfa, 2-processGraph/unitig-mapping-1.txt log: 2-processGraph/process.err jobid: 3 reason: Missing output files: 2-processGraph/unitig-unrolled-hifi-resolved.gfa; Input files updated by another job: 1-buildGraph/hifi-resolved.gfa, 1-buildGraph/paths.gaf resources: tmpdir=/tmp, job_id=1, n_cpus=1, mem_gb=64, time_h=24

[Mon Apr 1 10:07:21 2024] Finished job 3. 7872 of 7889 steps (99.8%) done Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-03-04T161008.807870.snakemake.log`

skoren commented 3 months ago

Snakemake should be smart enough to re-run any failed step if you resume. You can confirm it will run with the option --snakeopts --dry-run to verkko which will print the steps that need to run. You can also increase the time for this step from the default via --mbg-run 4 100 240 which should set the timeout to 10 days (assuming your grid allows that much time, if not adjust the last value to the max number of hours on your grid).

KANGYUlab commented 3 months ago

Thank you so much. There's a little question, should I run --snakeopts "--touch" before " --rerun-incomplete"?

skoren commented 3 months ago

It shouldn't be necessary to run touch but if --dry-run shows that there are jobs it's trying to run that have already been completed, then yes touch would fix that.

skoren commented 2 months ago

Should be resolved, please reopen if not.