Support generic stage labels

[samuelbernardolip]

As an improvement from previous release, JePL will support a special case for defining the composer responsible for the deployment and docker-registry operations on specific stages. For this a set of properties are expected for each one:

• by default will always run the following deployment to assure regression testing and backward compatibility

  deployment:
  composer: 'docker-compose'
  configuration: './.sqa/docker-compose.yml'

• docker registry support in specific stage

  docker_registry:
  services:
  - service1
  - service2
  options:
  ignore_push_failures: 'false'

The challenge here is to understand if it is possible to use any name for the property "deployment" and "docker_registry", without requiring specific prefix.

In the end, such deployment as docker_registry can appear in any position when defining the pipeline configuration, but with the requirement that the deployment only applies to the following projects and docker-registry always needs a previous deployment:

deployment0:
  composer: 'infrastructure manager'
  configuration: './.im/infra.tosca'

deployment1:
  composer: 'docker-compose'
  configuration: './.sqa/docker-compose.yml'

whatever-project-id1:
  sqa_criteria:
    qc_style:
      builders:
        - commands: [ ..]
        - tox: [...]
        - maven: [...]

docker_registry1:
  services:
    - service1
    - service2
  options:
    ignore_push_failures: 'false'

deployment3:
  composer: 'kubernetes'
  configs:
    - './k8s'
  options:
    template: 'false'
    kustomize: 'true'
    helm: 'false'
    apply: 'true'

whatever-project-id2:
  sqa_criteria:
    qc_style:
      builders:
        - commands: [ ..]
        - tox: [...]
        - maven: [...]

docker_registry2:
  services:
    - service1
    - service2
  options:
    ignore_push_failures: 'false'

[orviz]

@samuelbernardolip how are the services/deployments aligned with the 1) criteria and 2) registries? So far we have the container property (inside the criteria) to point to the service from the deployment. The deployment is unique so there is no need to refer to it. With this approach, we need to both define the service and the deployment, how do you expect to do that?

[samuelbernardolip]

how are the services/deployments aligned with the 1) criteria and 2) registries?

Building the required layers: 1- get the deployment with all the required services 2- run the criteria defined operations over the deployment 3- push the images to the registry

Step 3 don't depend on 2, but 2 and 3 depend on 1.

So far we have the container property (inside the criteria) to point to the service from the deployment. The deployment is unique so there is no need to refer to it. With this approach, we need to both define the service and the deployment, how do you expect to do that?

The container property is a hack in previous release to have this feature available as soon as possible. With this new approach all concepts become part of the configuration description and solves the limitation of the phantom docker operations stage in the end. With this proposal becomes possible to define the stage where desired.

The deployment have the current default using docker-compose.yml for DockerCompose composer. With other composers the default should also be defined and if not it should fail if required. So in the end, if a deployment stage doesn't exists, JePL will try the defaults as it happens in current release 2.1.0.

[samuelbernardolip]

@orviz as we review in our meeting the implemented solution will follow the next syntax:

configs:
 credentials:
  sadkfjlkas
 deployments:
  asdfasdfa1:
    composer: 'kubernetes'
    tools:
     apply:
      image: bitnami/kubectl:1.21
      repo: https://worsica.github.com/worsica/k8s.git
      filenames:
      - './k8s/confs'
      flags:
      - '--recursive'
     kustomize:
      image: bitnami/kubectl:1.21
      repo: https://worsica.github.com/worsica/k8s.git
      directory: './k8s/dev'
      flags:
      - '--force-conflicts'
     helm:
      image: alpine/helm:3.7.2
      repo: https://worsica.github.com/worsica/k8s.git
      values:
      - './k8s/helm/override.yaml'
      release: worsica
      chart: 'k8s/worsica'
      chart_repos:
      - 'https://worsica.github.io/helm'
      flags:
      - '--reuse-values'
     kpt:
      image: gcr.io/kpt-dev/kpt:v0.39.3
      repo: https://worsica.github.com/worsica/k8s.git
      path: './k8s/kpt'
      init:
      - '--name worsica'
      - '--namespace worsica'
      apply:
      - '--force-conflicts'
      - '--install-resource-group'
    run:
     pre:
     - podTemplate: https://github.com/worsica/cluster/im.yaml
       container: im
       commands:
       - im $sakldal
     - podTemplate: https://github.com/worsica/cluster/kubespray.yml
       container: kubespray
       commands:
       - ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
     post:
     - podTemplate: worsica/test:latest
       container: test
       commands:
       - test.py
 container_groups:
  groupA:
   services:
   - service1
   - service2
   options:
    ignore_push_failures: 'false'
    force_docker_build: 'false'

sqa_criteria:
 qc_style:
  repos:
   repoA_c1:
    deployment:
     name: asdfasdfa1
    service: c1
    keepgoing: true
    push: groupA
    commands: [ ..]
    tox: [...]
    maven: [...]
   repoA_c2:
    deployment:
     name: asdfasdfa1
    service: c2
    keepgoing: false
    commands: [ ..]
    tox: [...]
    maven: [...]
    push: groupA
 qc_delivery:
  repos:
   repoA:
    deployment:
     name: asdfasdfa1
     cleanup: true
    push: groupA
    keepgoing: true

There are other improvements to ignore failures by stages, besides complete pipeline, that will be not merged now. Issue #163, related to remove repos property, will not be implemented with current proposal. Generic labels approach whatever-project-id2 will be left for later review, keeping now with only two main properties as configs: and _sqacriteria: to pass regression testing on this phase. For the same reason, the support of builders: property will also be left here for later improvement, so JePL supports previous release configuration syntax and we can keep working on release 2.

[samuelbernardolip]

Add kops and kubespray to available deployment tools.