Mapping workflow

[daler]

This PR makes a bunch of changes (and supersedes #32 which is now closed).

Additions

• Adds workflows/mapping, which has many of the same testing infrastructure as test, like get-data.sh, config.yaml, Snakefile.
• workflows/mapping/Snakefile is a modified version of the one in test, with changes implemented as described below

  Testing

• You can test the mapping workflow by passing --workflow workflows/mapping to the now-updated run_test.py. The most complete test would be this (build a new env, clean all data, submit to slurm, test the mapping workflow):

test/run_test.py . --build-env --sbatch --clean --threads=8 --mem=32g --workflow=workflows/qc

• Using --sbatch will submit to the cluster for you, and you'll get an email upon completion

  Snakefile and config changes

• The references workflow is include:ed into the mapping workflow. Using subworkflows didn't work because I couldn't figure out how to pass a custom config file to a subworkflow.
• The contents of workflows/references/references_config.yaml have been moved to workflows/mapping/config.yaml in order to have a One True Config File (TM) in which all configuration occurs
• Along those same lines, there is a new rules section in the config. This is where you can edit the params for each rule (see the config for examples). Importantly, you can make a change in the config file, and then run snakemake --list-params-changes to identify those rules now out-of-date due to differences in params from the cache. You can then force these to re-run by using:

snakemake -f $(snakemake --lp) --configfile config.yaml

• The fastq_screen rule has been factored out into a separate rule that creates the fastq_screen_config.txt file. This in turn will build the bowtie2 indexes if needed. This works because the references snakefile is now include:d in the mapping workflow.
• I'm now pointing the references to a copy over in a references-test dir, so you can test index building without messing up the "real" indexes. I'd recommend starting by removing the wolbachia dir to see the effects (the fasta should be downloaded and the bowtie2 index should be created).
• The aligner has been generalized. See the align rule and the rules:align block in the config file for how this is configured. This requires that an aligner's wrapper accept fastq and index inputs, and output a BAM file, but I figured that was a reasonable requirement.

[jfear]

Tests worked!! Don't know if you want to go ahead and merge or if you want to think about some of my comments first.

[daler]

Let me address your comments first before merging, lots of things to fix. Thanks for reviewing.

[jfear]

Do you want me to do the tests and go ahead and merge, or do you want me to branch and hack?

[daler]

Hmm . . . test -> merge -> hack please

[jfear]

will do