Evolution for the CI / CD Workflows

[Jean-Baptiste-Lasselle]

This task is to :

• bring up a CICD tool chain in a k3d cluster, with inlets and inlets operator to be able to receive webhooks from github.com and gitlab.com, in my home installation of run atlantis, and drone .
• bring a major change to the drone pipeline, following the specs below

Pipeline Specs

• First, the https://github.com/pegasus-io/this-is-gravitee versions a recipe to provision gravitee.
• this https://github.com/pegasus-io/this-is-gravitee recipe, is a software component, and such, we want a pipeline for it (Infra As code), and that pipline will do this :
  • everytime we have a new commit ${GIT_COMMIT_ID}, the .drone.yml pipeline is executed
  • .drone.yml step - build the recipe (if needed) : .drone.yml step
  • .drone.yml step - will run some unit testing frameworks for :
  • shell : https://github.com/shellspec/shellspec
  • helm chart : https://github.com/lrills/helm-unittest https://github.com/helm/chart-testing
  • and ansible playbooks : https://molecule.readthedocs.io/en/latest/ and https://www.jeffgeerling.com/blog/2018/testing-your-ansible-roles-molecule
  • .drone.yml step - reset desired initial state for the deployment target :
  • When deployment target does not pre-exists, terraform is executed, creating the Virtual Box VM, and executing as terraform provisoners :
    • when it is a docker compose based deployment : an ansible playbook to install docker / docker-compose, and a docker-compose runing grafana, grafana loki, prometheus, and cAdvisor, to have a monitoring of the deployment target and the app inside during pipeline operations.
    • when it is a kubernetes based deployment: an ansible playbook to install docker and k3d, create the k3s cluster, export the KUBECONFIG, and store the kubeconfig in hashicorp vault. Also helm install all required components for the deployment of Gravitee : Traefik, Metlalb, External DNS PowerDNS. Also will execute helm install to deploy grafana, grafana loki, prometheus, and cAdvisor, to have a monitoring of the deployment target and the app inside during pipeline operations.
    • The terraform is executed by run atlantis, so this step will run git-flow commands and github/gitlab api commands, to trigger atlantis plan and atlantis apply.
  • When deployment target already exists, restoring initial state would mean :
    • if the deployment is docker-compose based : docker-compose down the previously deployed docker-compose, delete all folders mapped to docker volumes, delete all images, volumes, networks and containers, with docker system prune -f --all && docker system prune -f --volumes.
    • If the deployment was kubernetes/helm chart based, then I just need to helm delete the helm release. Can add an option to "prune, that is to say, fully delete an re-create the k3s cluster using k3d, and helm install all required components for the deployment of the software component (here a not just a software component, but all components of a software, namely Gravitee) : Traefik, Metlalb, External DNS PowerDNS, Mongo Replcaset for Gravitee. Also respawn the monitoring stacl grafana, grafana loki, prometheus, and cAdvisor.
  • .drone.yml step - run infrastructure tests
  • .drone.yml step - deploy gravitee : ansible playbook for docker-compose based, helm for a kubernetes based deployment. ansible playbook can be automated using Ansible AWX (Tower),and running the helm chart can be automated using fulxcd
  • .drone.yml step - run gravitee stack tests :
  • create api configure them and test the configuration, craete APi KEY, etc... automated test of the whole lifecycle of an API in Gravitee. done with a pod inside the cluster, and aerokube/selenoid /selenium Tests
  • create an API with API KEY in gravitee, then run a double galing perf test hitting the API through the GRavitee Gateway. Double, because the same GAtling test will be run once against the "bare" api (without Gravitee Gateway in between), and once with the gateway ther ein between . This way, we can evaluate the Gravitee overhead. The API that I will use here will be fully stateless, with traefik ingress route,service,replicasetanddeployment(100% scallable), will have just oneGET /sputnik` endpoint, which will simply return a JSON :

{
  "message" : "beep-beep-beep",
  "whoami": "<result of whoami command inside container processing the request>",
  "hostname": "<result of hostname command inside container processing the request>",
  "origin": "<value of the Origin HTTP Header in the request>"
}

Note there is one important part missing here : all test generate tests results, which should be collected, to generate a global tests results reporting dashboard.

The drone gravitee effect

I will deploy a private drone in my private kuberentes cluster at home :

• in the drone pipeline when they execute, steps will very often send gitab / github REST API HTTP requests,
• those reqiuest will all go through a gravitee API in my CICD tool chain : that way I can track ever single pipeline execution :
  • every pipeline has its own gravitee application,
  • for a given pipeline, two distinct executions of the same pipeline use dfifferent API KEYs, and in the gravitee porcelain, that means two different subsciptions of the same application, for the same subscribe plan.
  • then elasticsearch data will very quickly sort between every pipeline execution
  • configuraztion can specify how long before a complete reinit of the gravitee API gateway : cause this will have to happen on a reccurent basis, so that we do not push on disk pressure

Switch deployment targets

Well to do that, very simple : new deployement target, means new pippeline (no multyi target piplines, orchestrate pipelines instead)

So for example, I can quickly span a new pipeline that operates the deployment target :

• with the runatlantis at home, thanks to the inlets , but I could also use another run atlantis instance I have in a public cloud
• on AWS instead of my home VirtualBox :
  • needs 100 Gb of AWS EBS Elastic Block Storage (hard disk), 2 vCPUs, and 8 Gb RAM (will test if 4Gb is enough).
  • Don't forget to configure the AWS EC2 instance for elastic search requirements : to do hat use hashicorp packer to make a custom AWS AMI image with everything presinstalled as desired (PXEless boot)

[Jean-Baptiste-Lasselle]

• add JMeter taurus step : https://www.youtube.com/watch?v=BZ2Zm-SDGjI
• add vulnerabbility scan, more fine grained pipeline triggering events (see when clause below) and other features using example https://github.com/UKHomeOffice/cop-performance-tests/blob/master/.drone.yml :

pipeline:
  synch_dev_secrets:
    image: quay.io/ukhomeofficedigital/cop-secrets
    environment:
      - DRONE_SERVER=https://drone.acp.homeoffice.gov.uk
      - DEPLOY_ENV=dev
    secrets:
      - source: DEV_DRONE_AWS_ACCESS_KEY_ID
        target: AWS_ACCESS_KEY_ID
      - source: DEV_DRONE_AWS_SECRET_ACCESS_KEY
        target: AWS_SECRET_ACCESS_KEY
      - source: DRONE_PUBLIC_TOKEN
        target: DRONE_TOKEN
    when:
      environment: secrets
      event: deployment

  build:
    image: quay.io/ukhomeofficedigital/drone-docker
    commands:
      - docker build -t performance-tests:$${DRONE_COMMIT_SHA} .
    when:
      event: push

  vulnerability-scan:
    image: quay.io/ukhomeofficedigital/anchore-submission:latest
    dockerfile: Dockerfile
    image_name: performance-tests:${DRONE_COMMIT_SHA}
    local_image: true
    tolerate: low
    fail_on_detection: false
    when:
      branch: master
      event: push

  image_to_quay:
    image: quay.io/ukhomeofficedigital/drone-docker
    secrets:
      - source: QUAY_USERNAME
        target: DOCKER_USERNAME
      - source: QUAY_PASSWORD
        target: DOCKER_PASSWORD
    registry: quay.io
    repo: quay.io/ukhomeofficedigital/cop-performance-tests
    tags: ${DRONE_COMMIT_SHA}
    when:
      branch: master
      event: push

  deploy_to_dev:
    image: quay.io/ukhomeofficedigital/kd
    secrets:
      - source: DEV_API_REF_URL
        target: REFDATA_URL
      - source: DEV_KEYCLOAK_URL
        target: KEYCLOAK_URL
      - source: DEV_KEYCLOAK_REALM
        target: KEYCLOAK_REALM
      - source: DEV_KUBE_NAMESPACE_PRIVATE_COP
        target: KUBE_NAMESPACE
      - source: DEV_KUBE_SERVER
        target: KUBE_SERVER
      - source: DEV_KUBE_TOKEN
        target: KUBE_TOKEN
      - source: DEV_PERF_TEST_AUTH_CLIENT_ID
        target: AUTH_CLIENT_ID
      - source: DEV_PERF_TEST_AUTH_CLIENT_SECRET
        target: AUTH_CLIENT_SECRET
      - source: DEV_PERF_TEST_NAME
        target: NAME
      - source: DEV_PERF_TEST_POSTGREST_NAME
        target: POSTGREST_NAME
      - source: DEV_PERF_TEST_REFDATA_USERNAME
        target: REFDATA_USERNAME
      - source: DEV_PERF_TEST_REFDATA_PASSWORD
        target: REFDATA_PASSWORD
      - source: DEV_TESTS_REPORT_BASE_URL
        target: REPORT_BASE_URL
      - source: DEV_TESTS_SLACK_WEBHOOK
        target: SLACK_WEB_HOOK
      - source: DEV_TESTS_S3_KMS_KEY_ID
        target: S3_KMS_KEY_ID
      - source: DEV_TESTS_S3_ACCESS_KEY
        target: S3_ACCESS_KEY
      - source: DEV_TESTS_S3_BUCKET_NAME
        target: S3_BUCKET_NAME
      - source: DEV_TESTS_S3_SECRET_KEY
        target: S3_SECRET_KEY
      - source: DEV_WWW_REF_URL
        target: REFDATA_BASE_URL
    commands:
      - export IMAGE_TAG=${DRONE_COMMIT_SHA}
      - kd --insecure-skip-tls-verify -f kube/ref-data-small-dataset.yml
      - kd --insecure-skip-tls-verify -f kube/ref-data-large-dataset.yml
    when:
      branch: master
      event: [ push, deployment ]

  notify:
    image: plugins/slack
    secrets: [ SLACK_WEBHOOK ]
    channel: cop-deployments
    username: Drone Build Watcher
    template: >
      {{#build.deployTo}}
        *{{repo.name}} - Build {{build.number}} - {{uppercasefirst build.deployTo}} - {{uppercase build.status}}*
        {{build.link}}
      {{else}}
        *{{repo.name}} - Build {{build.number}} - Development - {{uppercase build.status}}*
        {{build.link}}
      {{/build.deployTo}}
    when:
      branch: master
      event: [ push, deployment ]
      status: [ success, failure ]

[Jean-Baptiste-Lasselle]

add automated test fro the CORS configuration of an API :

• deploy https://github.com/pegasus-io/the-cors-tester to the k8s cluster, on the docker-compose
• run the mocha chai tests to test the CORS
• generate the rest results report and publish it to team only accessible website, the project dashboard

[Jean-Baptiste-Lasselle]

Alternatives to FluxCD

• I want to try eunomia : https://github.com/KohlsTechnology/eunomia

[Jean-Baptiste-Lasselle]

Terragrunt Terratest and Atlantis

ok, so :

• I need to use atlantis with terragrunt, always, : https://www.runatlantis.io/docs/custom-workflows.html#use-cases
• and I want to test the terraform code with things like terratests
• So I will use drone to run all tests, including terratests, tests against a mocked Cloud API (like https://github.com/localstack/localstack ) , terraform linters, ansible linters, terraform validate, and terragrunt plan. Will eventually add a link to the full test report, inside the pull/merge request comments
• so sending the pull/merge request will trigger atlantis plan, so terragrunt plan

Atlantis has very interesting features for workflows, like the [mergeable] condition on a pull/merge requests.

Terragrunt helps coding with modularity in terraform code : important to have test friendly code (code that can easily be tested).