Feature | Testing: unit + integration + functional e2e CI automation

[exequielrafaela]

What?

• ✅ Repo CLI:

  • create automated unit tests; initially to run at the time of creating a PR; then we could evaluate if pre/post release as well.
  • create automated tests, integration & functional; initially to run at the time of creating a PR; then we could evaluate if pre/post release as well.
    • Test leverage credentials
    • Test leverage project create

• ✅ Repo Ref Arch: create automated, functional tests; that run on-demand and at the moment of releasing a new version of the CLI; it is necessary to define well which accounts and layers will be included since it can become very complex depending the test-cases.

  • terratest is the reference tool here, we use it to test the modules eg: https://github.com/binbashar/terraform-aws-waf-owasp/tree/master/modules/waf-global/tests
  • The thing is that we are going to need to put together something that focuses on testing the ref-arch but using the leverage-cli.
  • To start with I can think of some workflows that take certain layers to do the init, plan, apply and destroy workflow. @diego-ojeda-binbash OJ has already been working on something related -> https://github.com/binbashar/le-devops-workflows
    • Eg Nat Gw Disable: https://github.com/binbashar/le-devops-workflows/actions/workflows/shared_network_natgw_disable.yaml
    • Eg Nat Gw Enable: https://github.com/binbashar/le-devops-workflows/actions/workflows/shared_network_natgw_enable.yaml
  • But it is not strictly oriented to testing, still the demo-app envs creation through ref-arch with automated and parameterized workflows could help a lot here and guide us on how to automate e2e functional tests scenarios. In other words It can serve as a base or inspiration for these e2e functional bb-leverage Ref Architecture for AWS tests.

Why?

Having the proper level of testings

➕ Makes the Process Agile ➕ Quality of Code ➕ Finds Software Bugs Early ➕ Facilitates Changes and Simplifies Integration ➕ Provides Documentation ➕ Debugging Process ➕ Design ➕ Reduce Costs

---

@diego-ojeda-binbash OJ we'll wait for your review and feedback here in order to segregate this Testing Epic in the proper specific issues. Thanks in advance!

CC: @binbashar/leverage-ref-architecture-aws-dev @binbashar/leverage-ref-architecture-aws-admin

[angelofenoglio]

Proposal for testing Ref Arch Repo:

• Get all layers with changes: Get all changed files (maybe via GitHub API?) -> dirname -> uniq -> loop over
  • If layer has tests folder: Folder holds test file and .tfvars with expected values
    • Invoke bats

Layer directories can be concatenated and bats only invoked once

Bats test per layer:

Setup

1. Copy .tfvars to layer directory

   Test

2. leverage tf init
3. leverage tf plan -detailed-exitcode
4. If exit code is 0: 3.1. leverage tf apply 3.2. leverage tf output -json 3.3. Use jq and bats_assert to compare against .tfvars
   4. If exit code is not 0: 4.1. Post leverage tf plan output as comment to PR

Layers to test initially

• apps-prd/base-identities

[exequielrafaela]

@angelofenoglio @diego-ojeda-binbash the testing approach looks very promising. Couldn't wait to see the 1st workflow in place 🙌🏼 💪🏼

[angelofenoglio]

• Unit and Integration testing are considered done.
• Functional testing will be implemented as a simple GH action in Ref Arch repo over a dummy layer where init, plan, apply, destroy commands will be executed checking for no errors.

[exequielrafaela]

@angelofenoglio thanks for everything, @juanmatias will be collaborating with this task from now on 💪🏼

[exequielrafaela]

@juanmatias let's create another issue in order to test the leverage CLI dependency package binaries that could be found here:

• https://github.com/binbashar/le-docker-images/blob/master/terraform-awscli-slim/Dockerfile
• https://github.com/binbashar/le-docker-leverage-toolbox/blob/master/Dockerfile

Eg test-cases (1st document and in a 2nd iteration automte)

• levarage sso workflow
• aws eks kubeconfig automated generation (aws eks update kubeconfig)

CC: @angelofenoglio @diego-ojeda-binbash

[juanmatias]

Regarding the Leverage CLI / RefArch integration

Prev Situation

tests were not working

Curr Situation

tests will work as soon as PR are merged in both repos

• https://github.com/binbashar/le-tf-infra-aws/pull/444
• https://github.com/binbashar/leverage/pull/133

Notes

Things that have been done here:

• set param with quotes due it being take as an int literal in other case

  
        client_payload: '{"leverage_branch": "${{ github.ref }}"}'

• github app needed permissions for write in actions in order to trigger with dispatch here: https://github.com/organizations/binbashar/settings/apps/bb-le-cli-ref-arch-trigger/permissions permission: Actions (Workflows, workflow runs and artifacts) also in the repo (https://github.com/binbashar/le-tf-infra-aws/) this action has to be confirmed

• ref to branch in target repo added by default the action will take main while we are using master, so this line was added to the source trigger action:

  
        ref: master

• var scope fixed: from this

  
        github_token: ${{ steps.authentication.output.token }}

  to this

  
        github_token: ${{ steps.authentication.outputs.token }}

• in refarch action switched from private clone to public one:

  # from this
        git clone git@github.com:binbashar/leverage.git
  
  # to this
  
        git clone https://github.com/binbashar/leverage.git

• adding pipenv to the runner with this command in the target action

  
        if ! (which pipenv 2>/dev/null); then echo 'Installing pipenv...' && pip install pipenv && pipenv --python $(which python) ; else echo 'pipenv is already installed'; fi

• added a step to set the toolbox version: Set Leverage Toolbox version since tests are using a build.env already set in the refarch repo, there's a need of set the version based on the CLI version

• credentials for aws

  • credentials are stored like this:
  • dependant profile (the one which needs to assume the role) in the .aws/config file
  • source profile credentials are stored in .aws/credentials file the command aws configure set, when used to set the source profile, stores the access key id in the config file and then, for the source profile, stores the access key in config file and secret in credentials one. but when using terraform there is a sort of a bug, and both the access key and the secret must be stored in the credentials file for the source profile

• credentials for programmatic user access were reset

cc @exequielrafaela @angelofenoglio @diego-ojeda-binbash

[exequielrafaela]

@juanmatias as agreed we'll migrate all the test to the leverage cli repo instead of having them segregated in the ref arch one. Since this tests will give us a good testing baseline for the cli, we've also discussed about creating a new issue to start the discussions of the ref arch v1 specific minimum set of tests that could also help to have a better understanding of the different components versions (bringing more certainties for the current compatibility matrix that could be found in the doc).

CC: @diego-ojeda-binbash @angelofenoglio

[juanmatias]

A test was created in the Leverage CLI repo.

Now the test action has three jobs:

• unit test
• integration tests (make test-int)
• integration_tests_cli_refarch

this last job does the following:

• using leverage cli creates a project (step wich in turn will download refarch templates) and tries to init a layer
• pulling the refarch repo and with the cli test layer:
  • init
  • apply
  • destroy

Please, @exequielrafaela @diego-ojeda-binbash @angelofenoglio , check this PR https://github.com/binbashar/leverage/pull/136