Matrix Jobs

[jithine]

Context

Support building in multiple build containers with minimal job config. For example a library which needs to be built for both 7.4 & 7.5 versions of CentOS, rather than user defining 2 separate jobs for this, SD can abstract it out in a single job config, with multiple jobs fanning out for that single job configuration

Example conifguration

jobs:
   matrix-job:
      matrix: 
        - centos:7.4
        - cutom-name-7.5:
              image: centos:7.5
              environment:
                  FOO: BAR
        - image_key_in_template_images
      steps:
        build: ./build.sh

With above config Screwdriver execution system should create 3 jobs which runs in parallel. Name of the jobs should be matrix-job-centos:7.4,matrix-job-cutom-name-7.5 & matrix-job-redhat:8

Job matrix-job-centos:7.4 runs with image centos:7.4 Job matrix-job-cutom-name-7.5 runs with image centos:7.5 Job matrix-job-redhat:8 runs with image defined in the template for key image_key_in_template_images

Objective

1. Support matrix jobs.

[jithine]

Also if we can model passing different environment variables for each of the matrix job, then we can achieve composition of jobs at a pipeline scale.

[scr-oath]

I wonder if this could be be implemented something like a yaml preprocessor that could just connect a before and after job. Possibly giving option to create sd/noop jobs before and after or use existing ones by amending the joining job’s requires

[dwmarshall]

FWIW, I don't think there should be any processing of YAML.

Where this would have greater utility would be to be part of a template so that the template's users don't need to do anything particularly special. It's trivially easy to create separate jobs to achieve the matrix effect, so we should be aiming for something more expansive.

Instead of having to write this:

jobs:
  main-default:
    requires: [~pr, ~commit]
    template: ruby/gem
    image: default
    environment:
      FOO: alpha
  main-ruby26:
    requires: [~pr, ~commit]
    template: ruby/gem
    image: ruby26
    environment:
      BAR: beta
  main-ruby27:
    requires: [~pr, ~commit]
    template: ruby/gem
    image: ruby27
    environment:
      BAZ: gamma

I might prefer to write something like this (with an example of how I might get the same environment affect):

jobs:
  main:
    requires: [~pr, ~commit]
    template: ruby/gem
    images:
      - default
      - ruby26
      - ruby27
  main-default:
    environment:
      FOO: alpha
  main-ruby26:
      BAR: beta
  main-ruby27:
      BAZ: gamma

The environment setting part needs to be thought out more, not quite sure my off-the-cuff idea is a good one.