Explore the benchmarking of Tools as workflows

[tschaffter]

• Tools submitted for evaluation in past challenges are often dockerized command-line programs.
• Tools submitted for evaluation in the NLP Sandbox are dockerized API services.

Both programs and API services have limitations that we are capturing in #51.

This Epic aims to explore the implication of defining a Tool as a workflow. The motivation is that workflow are more and more used to perform biomedical tasks. Sage recently ramped up effort in that domain by settings up a Nextflow Tower.

In my mind, here are the few pros and cons of workflow:

Pros:

• Workflow steps can be run on several servers (horizontal scaling)
• Re-running a workflow doesn't always start from scratch but instead re-use cache
  • Useful during the development and application of a tool, less for benchmarking tools, unless several submitted tools share the same initial set of steps?

Cons:

• Myriads of workflow standard, no consensus yet (CWL is popular but interest in NF from the workflow community recently increased).
• Steep learning curves before developing and running workflows?

Tasks:

• Identify and compare the pros and cons of requesting (challenge) participants to submit tools as programs, API (micro)services and workflows.
• Explore how we could benchmark workflow tools
  • Where would the workflow tool submitted for evaluation live (e.g. dockstore, biocontainers, workflowhub)
  • How to run and evaluate the performance of the workflow

Resources

• Schemas for the GA4GH Tool Registry API
  • Idea: a workflow tool would live in a public workflow registry like dockstore, biocontainers, workflowhub. We would then use the TRS to get information about a tool submitted, including the information on how to run the tool.

[thomasyu888]

Here are some additions to the pros and cons list you have above for the workflow submissions:

Pros:

• Pipeline can be explicitly outlined
• Having explicit steps in your pipeline allows it to be more modular. For instance, theres the possibility of replacing certain steps of a pipeline with other tools.

Cons:

• Steep learning curve (Similar curve to API based submissions + Docker + GitHub in my opinion)
• Evaluation of workflows ( CAVATICA, Nextflow Tower, Terra ) - but currently no set system (This will be explored very soon )
• Having multiple steps to the workflow means there is possibly many Docker containers. How will we support private Docker containers?

[tschaffter]

Steep learning curve (Similar curve to API based submissions + Docker + GitHub in my opinion)

Not sure why Docker and GitHub are part of the equation. Isn't Docker required when developing portable workflows (required in the context of benchmarking).

Having multiple steps to the workflow means there is possibly many Docker containers. How will we support private Docker containers?

Do you mean private images? How is this linked to the number of containers to run?

[thomasyu888]

Not sure why Docker and GitHub are part of the equation. Isn't Docker required when developing portable workflows (required in the context of benchmarking).

Docker and GitHub are part of the learning curve because not everyone uses them. GitHub is added because if we have template workflows or template API repos for people to pull from, we basically add GitHub as a dependency. If people don't know how to use GitHub, that is new technology they have to learn. I think lots of times we think we are making lives easier for participants, but maybe we really aren't.

Do you mean private images? How is this linked to the number of containers to run?

Not necessarily. Lets say in a workflow you have 5 steps. Each of the steps use a different Docker images. Regardless of whether each image is public or private, we would need the ability to:

• access the images (if private)
• Cache the image somehow (technically even when someone specifies a specific Docker image + tag, you can still write over it)