This repository contains examples of scientific workflows. The goal of this repository is to write workflow components for mapping metagenomics reads that can be used on different architectures.
The repository is divided into subprojects, details of each below. Each MG-* project directory contains a Snakemake workflow.
There are git branches allegro
for the allegro cluster and redwood
for the redwood server (MG-1 and MG-3). Check the 80GB_benchmarks directory for benchmarking results on an 80GB reference database.
Prerequisites:
Start by running the raptor_data_simulation workflow. Set input parameters in simulation_config.yaml
.
cd raptor_data_simulation
snakemake --cores 1
Then set the search parameters in search_config.yaml
. To run one of the MG-* snakemake workflows:
cd MG-1
snakemake --use-conda --cores {e.g 8}
Other useful flags:
--force-use-threads
add this flag to force threads instead of processes in case each process takes too much local memory to be run in parallel snakemake --dag | dot -Tpng > dag.png
don't run any jobs just create a figure of the directed acyclic graphsnakemake --dryrun
don't run any jobs just display the order in which they would be executed.Simulating DNA sequences with https://github.com/eseiler/raptor_data_simulation.
Run this workflow before running any of the MG-* workflows. The data simulation parameters are set in simulation_config.yaml
. All MG-x workflows have a separate configuration file called search_config.yaml
where prefiltering and search parameters should be set.
This is a state of the art representative workflow for mapping metagenomic reads. The prefiltering and read mapping steps of MG-R are identical to MG-1. MG-R additionally contains a species abundance estimation step.
Steps of workflow:
This workflow is optimized to be run on a local system with large main memory and multiple threads. The large main memory is used when working with the IBF (at least 1GB) which has to be read completely into memory. The FM-indices, IBF creation and read mapping are done using 8 threads.
Steps of workflow:
This version of a metagenomics workflow aims to work around the constraint of having low memory. A hash table based approach is used instead of the IBF.
Steps of workflow:
This workflow uses the same tools as MG-1 but divides the FM-index creation and read mapping between multiple jobs (as opposed to one multithreaded job). This is useful for running the workflow on the cluster.
Steps of workflow: