Plugins: Testing / Validation and CI/CD

ccapell commented 2 years ago

Idea: Each plugin will require a validate script (called validate or startup) in the plugin folder. When BitOps starts, we iterate through each folder and run each script. It could be nothing more than a simple echo Plugin xyz found but this allows for more complex scripts if required.

These scripts can validate that the plugin is installed. It also allows for more complex checks when starting.

For Example, AWS might start by running something like printenv | grep ^AWS which gets the environmental variables for AWS. If there are variables, then we can do things like validate the user, or even calculate other values.

Other plugins might do something like terraform --version

Updated: Testing Checklist

[x] Env validation on plugin runtime
- [x] terraform
- [x] helm
- [x] ansible
- [ ] aws
[ ] DECIDE: cmd in install.sh vs test.sh vs testing framework
- [ ] enforce the standard across all the plugins
[ ] Check the artifacts' existence
[ ] CI/CD for each plugin
- [ ] PR run
- [ ] cron/weekly run
- [ ] based on latest plugins-base image
- [ ] idempotence

arm4b commented 2 years ago

Nice. Something like pre-start-hook script, but for the plugins?

arm4b commented 2 years ago

Looking closer, there are plugin env validation scripts already:

There is no one for AWS though: https://github.com/bitops-plugins/aws so @ccapell you can add it.

I think the approach needs more standardization so the other plugins would follow similar way enforcing the validation.

arm4b commented 2 years ago

Following the issue Custom image build reports success on a failed plugin installation #239, the plugins should have tests and CI/CD, ensuring that they could be built properly.

In the https://github.com/bitops-plugins repository, every plugin might have a testing pipeline that will verify the full installation end-to-end.

Full plugins testing picture

Run the full install.sh in the CI pipeline (each plugin repo)
- for a plugin repo PR: test.sh if the tools were installed and really work
- with the latest plugins-base docker image
User side: on plugin install (docker build)
User side: validate env on deploy/runtime - #224
Idempotence tests
- at least, re-running the plugin install twice shouldn't fail
- example: current installation of helm plugin fails if running twice
Run weekly on cron
- prevent build issues from happening on the users side
- catch them early on our side with testing
- because upstreams fail and install instructions change

Testing Framework

Option 1: Just add another command in the install.sh, like ansible plugin already does: https://github.com/bitops-plugins/ansible/blob/29af954b11406546160229f60a23967ee42029c4/install.sh#L44-L45 though it might be forgotten and not enforced.

Option 2: Plugins could just have another test.sh script with some simple instructions (like helm version)

Option 3: Perhaps plugins would need more testing cases and scenarios, then we could use https://github.com/bats-core/bats-core as a testing framework, considering plugins are already in bash to make it more organized:

#!/usr/bin/env bats

@test "kubectl is installed as a dependency" {
    run kubectl version --client=true
    assert_success
}

@test "helm binary is in the bin" {
    assert_file_exist /usr/local/bin/helm
}

@test "helm version as expected" {
    run helm version
    assert_success
    assert_line --partial 'Version:"v3.9.0"'
}

# example with conditionals
@test 'tiller with helm2' {
  if [[ $HELM_VERSION = 2.* ]; then
    skip "tiller is not installed"
  fi
  do_more_because_helm2
}

bitovi / bitops