Define the user story

[Adam-D-Lewis]

I think defining what usage looks like for the intended users will help us get on the same page. I think it's still evolving at this stage, but it'll be nice to document it here.

[Adam-D-Lewis]

@jerowe @dharhas As I understand it

The user:

• creates a workflow file containing:
  • resources for job (cores, threads, memory, walltime, etc.)
  • the tasks of the job itself as shell scripts
  • the names and dependencies of the tasks
• logs in to HPC on the login node
• installs miniconda
• creates appropriate conda environment
• runs a python script with the workflow file as a command line input, in which the script will:
  • read/parse the workflow file
  • provision dask workers via JobQueue
  • build task graph
  • execute tasks in task graph

I'm pretty sure ^^^ is not what y'all have in mind, but I'm still not clear on what it looks like so please point me in the right direction.

[jerowe]

Yes, this is exactly it, but these steps are optional:

• installs miniconda
• creates appropriate conda environment

If they are someplace with an admin or core team they may not install software themselves.

[jerowe]

@balast would it be helpful if I outlined a few of the common analysis types bioinformatics people tend to do and how they think about them? One thing I've realized is that coming from HPC I'm thinking of things differently than you all have been, and I'm trying to get everything straight in my head so I can translate.

[Adam-D-Lewis]

@balast would it be helpful if I outlined a few of the common analysis types bioinformatics people tend to do and how they think about them? One thing I've realized is that coming from HPC I'm thinking of things differently than you all have been, and I'm trying to get everything straight in my head so I can translate.

@jerowe I think that would be very helpful. I'm curious how large the data inputs/outputs to these workflows are. I'm particularly interested at a "per task" level.