Enable continuity i.e., End-to-End execution

[sreichl]

Goal: Once everything is done once i.e., every module was configured and executed, end-to-end execution should be possible for both, iterations with changed parameters and reproducibility.

• [ ] "straw man rule": Have one additional rule in dea_limma/dea_seurat that is only part of the DAG when the result files, the feature lists of one comparison/group, are requested by a subsequent module as in put (e.g., enrichment_analysis) otherwise the main aggregate rule generates all the files, without tracking them as output (because that's not possible).
  • [ ] https://github.com/epigen/dea_seurat/issues/16
  • [ ] https://github.com/epigen/dea_limma/issues/12
• [ ] docs: add how to use modules sequentially (also within recipes) If a module requires an annotation file generated by a previous rule, DAG construction fails if this file is missing. This can happen in project/recipe workflows where sequential module execution relies on outputs from preceding modules (e.g., an annotation file). The best practice is to run the workflow module by module, save the required output (e.g., the annotation file) externally, (e.g., to config/), and thereby making it re-runnable for future end-to-end execution for both, iterations with changed parameters and reproducibility. Checkpoints are no solution as they only apply to rules, not to modules loading data during DAG creation by Snakemake.
• [X] (not done, requires loading data for DAG creation) Potential solution?! One rule call per file. The feature list rule is its own rule that gets called for each feature list file (or group that always results in the same 3 files: up/down/Score) to be generated, even the empty ones (in parallel) -> how to implement annot? with an if that checks if the feature annot in config is filled? -> no checkpoints required. If I go ahead with this strategy (pre-determined outputs) I will introduce another limitation (hence the strong recommendation from the developers NOT to use data to configure your workflow), namely: we need to read out the metadata file for each dea_limma analysis (ie annot row) to determine the output files. but they might not exist yet. this currently works, because the metadata is not used for the workflow...

[sreichl]

• [ ] add as an example the ATAC-seq or spilterlize_integrate scenario, where both modules create annotation files for downstream analyses inputs.

[sreichl]

• [ ] How to use recipes extension For End-to-End analysis it is required that all config and annotation files exist. That means if you use the output of one module as annotation file for another (e.g., atacseq_pipeline's sample_annotation.csv as annotation file for spilterlize_integrate) you need to save a copy outside of the module's result folders (we recommend in the workflow's config/ folder of the respective dataset).