Proposal: Task arrays

[geoffjentry]

There was a discussion on the mailing list about providing task array functionality, which is a common feature among job schedulers

[buchanae]

One approach would be to set an environment variable describing the index in the task array. This variable would be available when evaluating Executor.cmd

[buchanae]

Related mailing list discussions: https://groups.google.com/a/genomicsandhealth.org/forum/#!topic/ga4gh-cloud/qFq_jgoRCvs https://groups.google.com/a/genomicsandhealth.org/forum/#!topic/ga4gh-cloud/ccaqjysBvZY

[jbingham]

FYI that we're in the processing of adding exactly this functionality into our dsub command-line, setting an env var for the task index. We're doing it client side, since our server impl doesn't support it server side.

[buchanae]

Ideas...

1) Task template + repeat count + loop index env. var.

• Sets an environment variable defining the current index in the task array for the executor to use.
• The loop index variable is not available to parts of the task message, such as inputs/outputs.

Roughly:

tpl = {
  "executors": [
    { "cmd": ["echo", "$TASK_INDEX"]},
  ],
  "inputs": [{
    "path": "/path/to/storage",
  }],
}

tes.CreateTaskBatch(tpl, repeat=1000)

2) Task template + repeat count + template variables

• Instead of an environment variable, the server must do variable substitution on the task's fields.
• Maybe want to keep template variable substitution as simple as possible, so that implementation is easy and bug free
• Still requires the user/client to map a task index to their data.
• Picking template delimiters that don't conflict with the rest of the world can be difficult.

Roughly:

tpl = {
  "executors": [
    { "cmd": ["echo", "{% TASK_INDEX %}"]},
  ],
  "inputs": [{
    "path": "/path/to/storage/{% TASK_INDEX %}",
  }],
}

tes.CreateTaskBatch(tpl, repeat=1000)

3) Task template + template variables list

• Like 2, but the user provides a list of variables to be substituted.
• More flexibility in how variables are used.

Roughly:

tpl = {
  "executors": [
    { "cmd": ["echo", "{% DRUG_NAME %}"]},
  ],
  "inputs": [{
    "path": "/path/to/storage/{% DRUG_NAME %}",
  }],
}

tes.CreateTaskBatch(tpl, vars=[
  {"DRUG_NAME": "foo"},
  {"DRUG_NAME": "bar"},
  ...thousands of rows here...
])

4) Task template + merging

• The user/client provides a base task template and a list of partial tasks. For each partial task in the list, the partial is merged with the base and a task is created from the result.
• Lists items are merged, not concatenated (e.g. Task.Executors, Task.Inputs, etc)
• Is merging too complex and possibly error prone?
• Probably the most heavyweight option, in terms of network bandwidth. How important is network bandwidth?

tpl = {
  "executors": [
    { "environ": { "shared": "foo" } },
  ],
  "resources": {
    "cpus": 10,
  },
  "inputs": [
    {
      "path": "/container/path",
    },
    {
      "path": "/container/path",
    },
  ],
}

tes.CreateTaskBatch(tpl, partials=[
  ... these are partial task messages, each row defining a specific override
  {
    "executors": [
        { "cmd": ["echo", "task1"] },
    ],
    "inputs": [
       { "url": "/path/to/task1/input1.data" },
       { "url": "/path/to/task1/input2.data" },
     ]
  },
  ... thousands of rows here ...
])

[buchanae]

This is a duplicate of #55 right? If so, I recommend we close #55