reanahub / reana-workflow-engine-snakemake

REANA Workflow Engine Snakemake
MIT License
0 stars 22 forks source link

utils: fix total job count to exclude restored jobs from workspace #64

Closed giuseppe-steduto closed 1 year ago

giuseppe-steduto commented 1 year ago

Amend the total number of jobs in a workflow so that when Snakemake detects that one or more jobs can be restored from the workspace (same workflow caching) they are not considered in the total count, reflecting Snakemake's local behaviour.

Closes #62.

How to test

roofit example

  1. Run the workflow normally with reana-client run -w roofit-snakemake -f ./reana-snakemake.yaml. The total number of jobs should be 2/2 (as usual)
  2. Restart the workflow in the same workspace with reana-client restart -w roofit-snakemake. The total number of jobs should be 0/0, because the workflow is entirely restored from the workspace.
  3. Change something in code/fitdata.C (such as a comment) and upload it to the workspace with reana-client upload -w roofit-snakemake code/fitdata.C
  4. Restart the workflow in the same workspace with reana-client restart -w roofit-snakemake. The total number of jobs should be 1/1, because only the fitdata step is executed (gendata is restored from the "local cache").

cms-h4l example

  1. Run the workflow normally with reana-client run -w cms-h4l. The total number of jobs should be 4/4 (as usual)
  2. Restart the workflow in the same workspace with reana-client restart -w cms-h4l. The total number of jobs should be 0/0, because the workflow is entirely restored from the workspace.
  3. Remove the Higgs4L1file.root (which is the input for the make_plot step, generated by analyze_mc) with reana-client rm -w cms-h4l-snakemake.1.4 results/Higgs4L1file.root and the final output plot: reana-client rm -w cms-h4l-snakemake.1.4 results/mass4l_combine_userlvl3.pdf
  4. Restart the workflow in the same workspace with reana-client restart -w cms-h4l. The total number of jobs should be 2/2, because the analyze_mc step should regenerate the Higgs4L1file.root file and the make_plot step should create the final output plot.
codecov[bot] commented 1 year ago

Codecov Report

Merging #64 (bc9cfee) into master (3a8f770) will not change coverage. The diff coverage is 0.00%.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/reanahub/reana-workflow-engine-snakemake/pull/64/graphs/tree.svg?width=650&height=150&src=pr&token=T60GN05T4Q&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub)](https://app.codecov.io/gh/reanahub/reana-workflow-engine-snakemake/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub) ```diff @@ Coverage Diff @@ ## master #64 +/- ## ====================================== Coverage 3.84% 3.84% ====================================== Files 6 6 Lines 182 182 ====================================== Hits 7 7 Misses 175 175 ``` | [Files Changed](https://app.codecov.io/gh/reanahub/reana-workflow-engine-snakemake/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub) | Coverage Δ | | |---|---|---| | [reana\_workflow\_engine\_snakemake/utils.py](https://app.codecov.io/gh/reanahub/reana-workflow-engine-snakemake/pull/64?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reanahub#diff-cmVhbmFfd29ya2Zsb3dfZW5naW5lX3NuYWtlbWFrZS91dGlscy5weQ==) | `0.00% <0.00%> (ø)` | |