🚇 Public App File Index Handling

[dmiller15]

Description

This PR addresses the changes necessary to make the workflow public app ready. Specifically these changes address the inability of having suggested secondaryFiles when running a public app.

The changes are made in the following ways:

• Added several tools to optionally perfom indexing (tabix, picard, bwa, samtools)
• Added a subworkflow that indexes the reference fasta and bundles those indexes as secondaryFiles
• Added a subworkflow that can untar a specifically designed tar that contains the reference fasta and its indexes
• Added additional tar tools that provide other tar utilities (listing and creation)

Changes to the workflow structure

• Added a step for dbSNP VCF index creation in the gVCF creation subworkflow
• Added input hook for dbSNP index to the gVCF creation subworkflow and propagated it out to the main workflow
• Input reference is changed from a file with secondaryFiles to a file (documentation and id indicate we are seeking a TAR for user clarity)
• Added steps in the main workflow to untar the reference tar, index as needed (in reality all of these steps will be skipped because the tar MUST have the files in it or the untarring process will fail). The result will be an indexed_reference_fasta which is used in all the same places the old one was
• Added a scattered step to index the items in the knownsites file list.

Resolves https://github.com/d3b-center/bixu-tracker/issues/661

Type of change

• [x] New feature (non-breaking change which adds functionality)
• [x] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• [x] Tool passes cwltool validation
• [x] Cavatica test: https://cavatica.sbgenomics.com/u/kfdrc-harmonization/kf-reference-pipeline/tasks/08705e71-0b4a-4811-9bbd-43997a706cf5/
• [x] Cavatica test of new knownsite index change: https://cavatica.sbgenomics.com/u/kfdrc-harmonization/kf-reference-pipeline/tasks/9442b46d-ca83-402d-90a3-7a672f41dfc2/

Test Configuration:

• Environment: cwltool 3.0.20200324120055, Cavatica
• Test files: See Cavatica test

Checklist:

• [x] My code follows the style guidelines of this project
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] I have made corresponding changes to the documentation
• [x] My changes generate no new warnings
• [x] I have added tests that prove my fix is effective or that my feature works
• [x] New and existing unit tests pass locally with my changes
• [x] Any dependent changes have been merged and published in downstream modules
• [x] I have checked my code and corrected any misspellings

[dmiller15]

Most recent test for adding flexibility to untaring and a fix to bwa indexing: https://cavatica.sbgenomics.com/u/kfdrc-harmonization/kf-reference-pipeline/tasks/0d96e7e4-d92a-44c8-9c5b-efe7e9c68602/stats/

[dmiller15]

Alright hopefully the last change to the fasta preprocessing. Keeping the tar unpacking but no longer supporting bwa, picard, or samtools indexing. Users will need to make their own. https://cavatica.sbgenomics.com/u/kfdrc-harmonization/kf-reference-pipeline/tasks/54a2b4b2-b928-4f13-96bc-31ffc3530a7f/stats/

[migbro]

If you look at the log in the test run, the command just seems to untar the whole thing wach time: tar xf /sbgenomics/Projects/0154a615-2b20-4c32-9c76-141b5eebdde1/Homo_sapiens_assembly38.tgz from https://cavatica.sbgenomics.com/u/kfdrc-harmonization/kf-reference-pipeline/tasks/54a2b4b2-b928-4f13-96bc-31ffc3530a7f/stats/ I thought it was supposed to grab a specific file?

[dmiller15]

I changed the way the tar process works. The single file untar process was only working when I used exact file names so I just decided to untar everything and sort it out with the glob.