Automate extracting release and version information from repo manifest

[gberche-orange]

Is your feature request related to a problem? Please describe.

Background

The cf-deployment model is bringing an independent, consistently/tested deployment that includes a set of releases and their versions.

https://github.com/cloudfoundry/cf-deployment/blob/9ed502c778e4335f0d4e78ba22d728b58a7bfa39/cf-deployment.yml#L1614-L1722 now includes 26 distinct releases.

As a result of important investments in CF release engineering team, there is a high rate of cf-deployment and associated releases refactorings and bumps. The cf community leverages this important release engineering effort.

Problem description

In order to leverage COA support for automated bosh release support (internet access standardization, bosh offline replication and upload to the bosh director), COA authors have to manually extract the bosh releases and their versions into two files:

• deployment level deployment-dependencies.yml file with list of releases and their source locations
• xx--root-depls-versions.yml files for versions.

This creates creates repetitive error prone work, in particular when release get shared among multiple deployments and may conflict within their root deployment, or feature branches to update xx--root-depls-versions.yml conflict

This was also reported by the service author team (/CC @plafosse-orange )

Describe the solution you'd like

A way for COA authors to not have to perform manual work in order to leverage a community deployment repo such as cf-deployment, while still leveraging COA automated release offline replication downloagin A way to automate the extraction of the releases and their versions from a community repo

Describe alternatives you've considered

• not leveraging COA bosh release support (i.e. leave the bosh release section of the manifest)

Additional context Add any other context or screenshots about the feature request here.

[gberche-orange]

Summary of discussion with @o-orand, we need to resivit the initial requirements/assumptions that drove the current bosh release support in COA (w.r.t. new bosh features that came up since):

• when do we need to enforce consistent versions bosh release version common across a root deployment ?
  • bosh add-ons provide an easier solution
• can we address the outgoing internet connectivity (explicit proxy vs implicit proxy) while leveraging bosh director bosh release manifest section ?
• can we address the bosh release caching while leveraging bosh director bosh release [manifest section] (http://bosh.io/docs/manifest-v2/#releases) ?
• how scalable are our bosh release offline replication pipelines (with 20, 40 bosh releases resources ?)

[poblin-orange]

Some thought:

• direct download from bosh director should be / will be forbidden (we will soon disable SNAT to internet).
• not all bosh release authors do provide a sample manifest. Most do not yet.
• paas-template deployment specific tooling could easily fix the COAB authors issue
• main focus of the current concourse lead versioning was to help operators manage global platform versioning. Have to avoid complete paas-templates bosh releases versions fragmentation

[gberche-orange]

not all bosh release editors do provide a sample manifest. Most do not yet.

List of current releases that do provide sample manifest in the form of a dedicated repo, or subdirectories

• https://github.com/cloudfoundry/cf-deployment
• https://github.com/cloudfoundry/bosh-deployment
• https://github.com/concourse/concourse-bosh-deployment
• https://github.com/cloudfoundry/cf-mysql-deployment
• https://github.com/cloudfoundry-incubator/kubo-deployment
• https://github.com/cloudfoundry-community/docker-broker-deployment
• https://github.com/bosh-prometheus/prometheus-boshrelease/tree/master/manifests
• https://github.com/pivotal-cf/credhub-release/tree/master/sample-manifests
• https://github.com/pivotal-cf/cf-rabbitmq-release/tree/master/manifests
• https://github.com/starkandwayne/shield-boshrelease/tree/master/manifests
• https://github.com/cloudfoundry/nfs-volume-release/tree/master/operations
• https://github.com/cloudfoundry/routing-release/tree/develop/templates (as a replacement of deprecated https://github.com/cloudfoundry-community-attic/route-registrar-boshrelease/tree/master/manifests)

Other releases that provide sample manifest but no yet operators (and which we could contribute on some of them):

• https://github.com/minio/minio-boshrelease/tree/master/manifests
• https://github.com/cloudfoundry-incubator/haproxy-boshrelease/tree/master/manifests
• https://github.com/pivotal-cf/cf-redis-release/tree/master/manifest
• https://github.com/cloudfoundry-community/ntp-release/tree/master/manifests
• https://github.com/orange-cloudfoundry/mongodb-boshrelease

Other releases that don't yet provide sample manifests:

• https://github.com/orange-cloudfoundry/metabase-boshrelease
• https://github.com/cloudfoundry-community/bui-boshrelease
• https://github.com/cloudfoundry-community/memcache-release