Workflow invocation fails when collections input is selected at runtime

[jennaj]

Three related issues: None have been added to Roadmap for triage yet.

BUG Most important item to fix.

• [ ] When workflows do not contain the optional Inputs dataset, and the input is a collection selected at runtime, the workflow is launched successfully, but no jobs are actually launched.

USAGE ENHANCEMENTS found during testing of the bug

• [ ] Some tools do not allow more than individual dataset entry at runtime. The workflow editor does not have the input-type option included as a tool parameter in the workflow editor. FastQC is one example. This means that Inputs are required if multiple datasets or a collection are to the input for at least one tool (a very common tool).

• [ ] Some tools do not accept Collection Inputs by default. The noodles do not connect. The collections type is stated as being "list" in free-form text for the tool options in the workflow editor. The user has to know to not only click into the "type" field but to start entering text in order for the pull-down menu of choices to show up. Example: BWA-MEM. This means it is not possible to use BWA-MEM and any other tools impacted as the first step in a workflow when a collection is an input unless you already know what you are doing. Maybe this should be a required selection and not automatically set as "type == list"???

ping @guerler @jmchilton

Fixes

• Requiring Inputs is one choice - fixing tools to allow collection runtime selection is another.
• Collection Inputs needs a better UI and some help text within the editor for setting "Type".
• Figure out why changing the Collection Inputs type more than once breaks the noodle connections to downstream tools.

Testing overview and links to test workflows and result histories

Datasets used as run-time inputs:

• https://usegalaxy.org/u/jen/h/test-history-paired-coll-human-dna

Workflows and result histories:

Collection input not connecting:

• https://usegalaxy.org/u/jen/w/test-wf-inputs---coll-select-with-inputs---wont-noodle-connect---bwa-mem

Collection input connecting when "type" is reset ONE TIME:

• https://usegalaxy.org/u/jen/w/test-wf-inputs---coll-select-with-inputs-set-with-type-list-paired---bwa-mem---works

Collection input NOT connecting when "type" is reset MORE THAN ONE TIME (bug):

• https://usegalaxy.org/u/jen/w/test-wf-inputs---coll-select-with-inputs-set-with-type-list-paired---bwa-mem

Only individual datasets can be selected at runtime (no multiple, no collection):

• https://usegalaxy.org/u/jen/w/test-wf-no-inputs---only-indiv-select-available-at-runtime---fastqc

Collection or individual datasets selected at runtime:

• https://usegalaxy.org/u/jen/w/test-wf-no-inputs---indiv-select-at-runtime---bwa-mem

• https://usegalaxy.org/u/jen/h/test-wf-no-inputs---indiv-select-at-runtime---bwa-mem

• https://usegalaxy.org/u/jen/w/test-wf-no-inputs---coll-select-at-runtime---bwa-mem-on-5

• https://usegalaxy.org/u/jen/h/test-wf-no-inputs---coll-select-at-runtime---bwa-mem-on-5

• https://usegalaxy.org/u/jen/w/test-wf-no-inputs---indiv-select-at-runtime---bowtie2

• https://usegalaxy.org/u/jen/h/test-wf-no-inputs---indiv-select-at-runtime---bowtie2

• https://usegalaxy.org/u/jen/w/test-wf-no-inputs---coll-select-at-runtime---bowtie2-on-5

• https://usegalaxy.org/u/jen/h/test-wf-no-inputs---coll-select-at-runtime---bowtie2-on-5

Graphic of History list for indiv vs coll select at runtime:

[bernt-matthias]

I just got a report from a user that seems to be equal to the 1st issue. Would be interesting to know if there is a workaround for the mean time.

[jennaj]

@bernt-matthias The solution is to use an Inputs within the workflow and set it to the correct collection type the downstream tool is expecting. Configure that downstream tool for the expected input type before as needed. Then the noodles will connect and the workflow will run normally.

Selecting collections as input at runtime is the core issue we will be addressing. Making usage more clear will be secondary priorities (enhancements) and are more complex to address but the help above should get your workflow going. In short, use Inputs, and configure them correctly, instead of selecting inputs at runtime. The user still has a choice of which actual input datasets to choose - the Inputs just defines which datasets are appropriate to use.

A single dataset will have one Inputs (can be a single dataset or a dataset collection).

Multiple datasets that are not in a collection will have multiple Inputs.

Any of the tool/steps can be annotated to guide the user in making proper choices. I often label these with the datatype and sometimes the target genome (if that is already set for some of the tools).

Example1: If a tool has a specific target genome already set, and the input is a BAM dataset, I'd label the input with "BAM for dbkey" where "dbkey" is the genome (hg38, mm10, some custom build, etc).

Example2: If a there are forward and reverse reads, I would include that info in the Inputs annotation as well, so the correct file is selected.

[jennaj]

@jmchilton @guerler @dannon Did we decide that using an Inputs within the workflow for collections is now required? And it will be that way going forward?

If so, we help users that encounter this issue as it comes up.

The other fixes could be considered enhancements. We could move those to a ticket specifically for "New" -- Should I do that and close this out?

[jennaj]

Related ticket: https://github.com/galaxyproject/galaxy/issues/7431

An enhancement for workflows should still probably be added. Users do not understand why there is no output from a workflow when no "input" is included at the start, in particular, a collection input. The workflow launches but doesn't produce any output. Is reported several times a week now.