Automated Testing of AWS Templates

[DinaBelova]

Goals

• Ensure that we deliver functioning templates to customers which cannot cause any outages or other issues when they use them to deploy clusters

Major deliverables

• All Templates that are provided by project are automatically tested and verified with actually deploying them to the infrastructure providers. This happens every time we create a new version of a template or a new version of project itself. Per-commit testing is not required for all tests, test that take a long time can be done on a release/tag level.

Who it benefits

• Business: Less outages and downtimes of applications
• Platform: Reduction of urgent issues
• Mirantis: Less angry customers

Acceptance criteria

• All existing AWS templates are fully automated tested against real world infrastructure
• All testing happens in Github
• Verification of cluster successfully being created: we can rely on CAPI/Infra Providers returning to us that the clsuter has been successfully created
  • Also verify working of additional helm charts installed into the clusters

Assumptions

• It is actually possible to test against real world infrastructure without massive amount of costs or engineering time involved to do so.

Limitations

• Budget for test infra

Out of scope

• Infrastructures covered by CAPI but not provided with Templates
• Upgrade testing, which will be covered in the relevant implementation epic

Related issues:

• 211

• 212

[DinaBelova]

Note: fell out from sprint-6 to sprint-7 due to couple of delays around certificates / permissions mgmt for CI as well as hosted vs standalone tests specifics. Is planned to be closed early sprint-7.