mikefactorial commented 3 years ago

Is your feature request related to a problem? Please describe. Need an automated way to perform integration tests on pipeline updates Describe the solution you'd like We should be able to create Pipelines that test the import, export, delete, build-deploy pipelines with triggers associated with PRs into the vnext branch of the ALM Accelerator Templates.

AB#1367

Grant-Archibald-MS commented 2 years ago

Overview

To support integration testing was to move towards shift left testing of Azure DevOps Pipelines:

Level 0 Tests (In Memory) - Investigate PowerShell Yaml to ensure properly formatted yaml files locally
Level 2 Test (What if tests with Azure DevOps) - Investigate Azure DevOps Preview YAML to validate pipeline changes as part of GitHub pull requests
Level 3 (Environment Tests) - Provide end to end integration tests. To enable these tests the following will be required
- Azure Active Directory - Azure Active Directory Application as Service Principal, Azure Active Directory Groups
- Azure DevOps - Azure DevOps Organization with
  - Extensions Installed
  - Service Connections to Power Platform Environments
  - Azure DevOps permissions to Build Agent
  - Azure DevOps permissions to System Accounts
  - Install Azure DevOps pipeline templates
  - Configured Azure DevOps Pipelines and Build Agents

GitHub Actions

GitHub actions will be required to the ALM Accelerator templates that will run tests to supporting the following cases:

Validated Pull Requests using L0 tests, L2 tests against configurable Azure DevOps Service
L3 Tests that will run in manual or nightly builds that execute end to end tests and configured service

Level 3 Testing

To support efficient execution of the end-to-end integration tests the following will be investigated

Build Agent

Add local build agent so that pre-requits components already exist on the build agent to speed up execution of test results investigate Self-Hosted Build agent

Given the ALM Accelerator currently uses a Windows Build agent investigate using Self Hosted Windows Docker Image to reproducibly test the pipeline and dependencies.

Environment Testing

To perform L3 test the following specific approach is proposed:

A set of behavior driven Cucumber tests that express scenarios and expected test results
Cumber support scripts that provide common actions to enable test scenarios. For example
- Delete Solution
- Import Solution
- Integrate with Azure DevOps services to:
  - Create new Azure DevOps project
  - Uses the coe cli to configure the project with a nominated pipeline branch
  - Execute individual pipeline tasks
  - Execute End to end pipelines
- Import ALM Accelerator Sample Application

Test Scenarios

The following test scenarios will be required:

Import Unmanaged Solution into Azure DevOps
Deploy solution to validation environment (Initial/ Upgrade)
Deploy solution to test environment (Initial/ Upgrade)
Deploy solution to production environment (Initial/ Upgrade)
Delete a solution
Individual tests for each Template

mikefactorial commented 2 years ago

Closing this out as it's been addressed in the July release as far as an MVP of end to end testing. Will open more specific issues for gaps

microsoft / coe-starter-kit