reanahub / reana-workflow-engine-snakemake

REANA Workflow Engine Snakemake
MIT License
0 stars 22 forks source link

executor: support custom resources #9

Closed mvidalgarcia closed 3 years ago

mvidalgarcia commented 3 years ago

Custom resources are set in Snakefile via resources hint.

closes #8

To test:

# reana-demo-wordpopulation

diff --git a/workflow/snakemake/Snakefile b/workflow/snakemake/Snakefile
index 05fdfe9..0bf124d 100644
--- a/workflow/snakemake/Snakefile
+++ b/workflow/snakemake/Snakefile
@@ -29,7 +29,9 @@ rule worldpopulation:
         output_file=config["output_file"]
     container:
         "docker://reanahub/reana-env-jupyter:2.0.0"
+    resources:
+        kubernetes_memory_limit="300Mi"
     shell:
-        "mkdir -p results && papermill {input.notebook} /dev/null -p "
+        "sleep 30 && mkdir -p results && papermill {input.notebook} /dev/null -p "
         "input_file {input.input_file} -p output_file {output.output_file} "
         "-p region {params.region} -p year_min {params.year_min} -p year_max {params.year_max}"

Run the example and check k8s job description via kubectl describe pod to verify that memory limits are properly set.

audrium commented 3 years ago

Tested kubernetes_memory_limit and it works as expected :+1:

...
Limits:
  memory:  300Mi
...
mvidalgarcia commented 3 years ago
* Shouldn't we also address this [FIXME](https://github.com/reanahub/reana-commons/blob/bad177fffc332c72440f310a77fc2e3d6b6f69ef/reana_commons/snakemake.py#L68) as part of the solution?

Yes, I've created a PR, thanks!