Add RNASeq autoscaling and register on Dockstore

As a researcher I want to run pipelines that autoscale and thus take advantage of multi-node parallelism

Tasks

[x] Review Toil documentation for autoscaling #118 #111
[x] Create proof of concept for autoscaling using Consonance VM to launch RNA-Seq which launches leader
[x] Write document outlining implementation options and review #50 https://docs.google.com/document/d/16FRkQWarCwgOjQkpRuzTqq1rqG5adv9XydzTjevEBrs/edit?ts=5a216dad
[x] Write initial implementation document https://docs.google.com/document/d/1VpfP-JbKw57qOczWzrw7FajBlBoNiYu1DKXepS0WBuw/edit

Action Service #51

[x] New parameter to indicate autoscaling
[x] Code to add multiple sample files to the parameterized json for input to the RNA-Seq pipeline
[ ] backport code to add multiple samples to job from script to action service

Testing #54

Invoicing #93

[ ] CGP will see the multiple samples as one job and calculation of compute cost will be for all the samples processed; the invoicing calculations will have to report an average compute cost per sample (how will we tell it how many samples were processed?)

Toil

[ ] Make fix to Toil so VM name does not have to be the same as the security group name for autoscaling
[ ] Make fix to Toil so that pipeline container /leader does not have to be named 'toil-leader' or make container launch with name 'toil-leader' for autoscaling
[ ] Enhance Toil so that it reports to user when security group settings are not correct for autoscaling

RNA-Seq pipeline (or other pipeline) Dockerfile #122

Wrapper script #52

[x] New parameter to indicate autoscaling
[x] Parameter for URL (S3 bucket) to store results
[ ] Code to create manifest for multiple samples
[x] Code to launch mesos leader
[x] Refactor code to accommodate RNA-Seq refactor so no concatenation is needed and files are pulled directly to pipeline using Redwood signed URLs

CWL #53

Testing #129

Dockstore tool runner #30 CWL

wrapper script

[ ] Parameter for output URL (S3 bucket)
[ ] Code to convert input file path to signed AWS (https) URL
[ ] Code to download results from output URL (S3 bucket) Or code to upload directly from output URL (S3 bucket) to Redwood storage system

Testing #31

Other Pipelines

BD2KGenomics / dcc-ops