Use a reference data set to increase test coverage

[o-orand]

Use the reference data set (#111) to enhance acceptance test. It is going to increase our test coverage.

A reference data set is composed by :

• a template repository (like Paas Template)
• a config repository

To run acceptance tests, we also need some prerequisites:

• a concourse instance, to deploy COA generated pipelines
• an S3 compliant, to host stemcells
• a bosh director: to be able to deploy new deployments or to test cloud/runtime config
• to be completed

After tests execution, we also need a way to assert concourse work.

Global overview:

[o-orand]

To validate pipeline (ie jobs and tasks execution), we can use the config repository. Indeed, most of the jobs generate files committed to config repository. So we can scan config repository or wait an expected time if files are presents ,and we may parse generated files to ensure files match our expectations.

[o-orand]

TODO:

• [x] generate per-root-deployment pipelines from reference dataset(already done by acceptance tests)
• [x] load singleton pipeline
• [x] load per-root-deployment => should be done by bootstraps-all-init-pipelines. Need to unpause.
• [x] iterate to enrich automatic validation for per-root-deployment pipelines
• [x] iterate to enrich automatic validation for singleton pipelines
• [x] cleanup all loaded pipeline (on manual approval)

Pipeline list: singletons => required by per-root-deployment-pipeline

• [x] bootstraps-all-init-pipelines: bootstraps all init pipelines
• [x] control-plane
• [ ] sync-feature-branches: manages a WIP branch based on develop and any feature branches
• [ ] ( sync-hotfix-branches: manages a WIP branch based on master and any hotfix branches)

per-root-deployment:

• [x] update/init-pipeline: initializes all pipelines related to a root-deployment
• [x] concourse-pipeline
• [ ] (news-pipeline: notifies of new bosh release versions for a root-deployment)
• [ ] sync-helper-pipeline: to ease secret repo management
• bosh pre-requisite
  • [x] bosh-pipeline/depls-pipeline: manages bosh deployments associated to a root-deployment
    • [ ] [runtime|cloud]-config
    • [x] terraform *
    • [x] deploy.sh
    • [ ] delete*
    • [ ] recreate*
    • [x] deploy.sh
    • [x] cancel
    • [x] init
• CF pre-requisite (we may use PWS)
  • [ ] cf-apps-pipeline: manages cf-apps associated to a root-deployment
• S3 pre-requisite
  • [ ] s3-br-upload-pipeline: to upload bosh releases used by this root deployment to S3
  • [ ] s3-stemcell-upload-pipeline: to upload stemcells used by this root deployment to S3

[o-orand]

@lucaspinto remaining TODO:

Pipeline list: singletons => required by per-root-deployment-pipeline

• [ ] sync-feature-branches: manages a WIP branch based on develop and any feature branches
• [ ] ( sync-hotfix-branches: manages a WIP branch based on master and any hotfix branches)

per-root-deployment:

• [ ] news-pipeline: notifies of new bosh release versions for a root-deployment
• [ ] sync-helper-pipeline: to ease secret repo management
• bosh pre-requisite
  • [x] bosh-pipeline/depls-pipeline: manages bosh deployments associated to a root-deployment
    • [ ] [runtime|cloud]-config => we should adapt our sample apply current cloud and runtime config
    • [ ] delete* => unpause
    • [ ] recreate* => unpause
• CF pre-requisite (we may use PWS)
  • [ ] cf-apps-pipeline: manages cf-apps associated to a root-deployment => this should be done in #234
• S3 pre-requisite
  • [ ] s3-br-upload-pipeline: to upload bosh releases used by this root deployment to S3 (already loaded)
  • [ ] s3-stemcell-upload-pipeline: to upload stemcells used by this root deployment to S3 (already loaded)