Well defined, delineated test responsibilities will clarify the CSM testing strategy. A clarified testing strategy will aid in quicker cycle times for developers, increased confidence in the quality of our systems, and in increasing the diagnosability of our systems.
Problem Statement
CSM has a large body of testing. These tests overlap responsibilities and roles. Sometimes these tests have less than ideal levels of coupling. This document will strive to delineate between the types of testing, what guarantees they make, and where our testing effort should be spent.
The way our current (CSM 1.2) functional test suites work is a hold-over from the 'unified' testing framework patterned when CSM was still part of the joint Devops community. 'smoke' and 'functional' tests are installed onto the NCNs that execute against functional areas of system management based mostly on team functional area.
The tests are delivered in a series of 'home-grown' bash and python scripts. They are installed via RPM. The tests are usually out of date because they are baked into the NCN images. We historically don't like to spend the time to re-create the NCN images, so the tests languish.
Internal References
Example test execution flow for CSM 1.2:
External References
Firstly I will submit there are many reasonably well aligned testing definitions:
These documents parse testing along several key axis:
type of testing based on the component scope of the test
i.e. are we testing just a single code path, single component, or some interaction between components
test execution actor (manual or automatic)
i.e. are the tests executed automatically, or what level or manual interaction do they require
test purpose - test the 'technology' vs test the 'story' the solution provides (e.g. user acceptance testing
i.e. are the tests functional use case driven? e.g. validate that actor can accomplish task? vs 'verify function input -> Process == expected output
test location
i.e. does the test run in a pipeline, developer environment, or on the real system
Proposed Solution(s)
While we could define dozens of very specific types of tests, this will lay out a select subset based on the nature of our work. This document will try to make concrete, the abstract possibilities for testing and clarify the expectations we have for our product.
Scope of Testing Philosophy
CSM is very broad. It has a great number of facets and components that accomplish a large set of use cases.
For the purpose of this document we will define CSM and the scope that the testing philosophy will be limited within
Defining CSM
CSM is a set of capabilities exposed via APIs,
that are
comprised from loosely coupled microservices,
HTTP based,
and RESTful,
that integrates and manages disparate
types of hardware (controllers, nodes, switches, etc),
from many vendors
Scope
This document will focus on the testing of the CSM APIs. Any component that is not an API should be tested, but is a separate concern from this document. This document will expressly not comment on testing related to:
manual processes and workflows
scripts and other low level utilities not exposed via an API
other user interfaces and experience modalities like the CLI
Required Testing
(TODO for each define)
unit testing
integration testing
functional testing
smoke testing
hardware testing
workflow testing
Tools and methodology
ARGO?
Helm Tests
give examples of the hardware tests
Briefly describe the main changes to CSM software involved in solving the problem. If there are multiple paths under consideration, feel free to list them. Links to external design documents are fine, but the level of detail needed is low. We need to understand what is being proposed. We do not need enough detail to implement.
Impact of Action/Inaction
What if we don't solve this problem at this point?
What impact is there beyond the problem statement if we fix the problem now?
Further Information
Use this section for pertinent information and/or links to background information.
Suggested Reviewers
@mention individuals who you would like to have review this proposal. All are welcome to review and comment, but SMEs or others who should be aware of this EP should be listed here.
- [ ] @reviewer1- [ ] @reviewer2
Comment Period
Specify a date by which this proposal's comment period should close. In order to ensure that lack of review does not delay mission critical enhancements, the suggested comment period is at least 7 calendar days, not including major holidays/breaks. Use your best judgment. If this is not time critical specify a longer comment period or none at all.
Comment period for this proposal shall close on [[July 28, 202X]].
Abstract
Well defined, delineated test responsibilities will clarify the CSM testing strategy. A clarified testing strategy will aid in quicker cycle times for developers, increased confidence in the quality of our systems, and in increasing the diagnosability of our systems.
Problem Statement
CSM has a large body of testing. These tests overlap responsibilities and roles. Sometimes these tests have less than ideal levels of coupling. This document will strive to delineate between the types of testing, what guarantees they make, and where our testing effort should be spent.
The way our current (CSM 1.2) functional test suites work is a hold-over from the 'unified' testing framework patterned when CSM was still part of the joint Devops community. 'smoke' and 'functional' tests are installed onto the NCNs that execute against functional areas of system management based mostly on team functional area.
The tests are delivered in a series of 'home-grown' bash and python scripts. They are installed via RPM. The tests are usually out of date because they are baked into the NCN images. We historically don't like to spend the time to re-create the NCN images, so the tests languish.
Internal References
Example test execution flow for CSM 1.2:
External References
Firstly I will submit there are many reasonably well aligned testing definitions:
These documents parse testing along several key axis:
Proposed Solution(s)
While we could define dozens of very specific types of tests, this will lay out a select subset based on the nature of our work. This document will try to make concrete, the abstract possibilities for testing and clarify the expectations we have for our product.
Scope of Testing Philosophy
CSM is very broad. It has a great number of facets and components that accomplish a large set of use cases.
For the purpose of this document we will define CSM and the scope that the testing philosophy will be limited within
Defining CSM
Scope
This document will focus on the testing of the CSM APIs. Any component that is not an API should be tested, but is a separate concern from this document. This document will expressly not comment on testing related to:
Required Testing
(TODO for each define)
Tools and methodology
ARGO? Helm Tests give examples of the hardware tests
Briefly describe the main changes to CSM software involved in solving the problem. If there are multiple paths under consideration, feel free to list them. Links to external design documents are fine, but the level of detail needed is low. We need to understand what is being proposed. We do not need enough detail to implement.
Impact of Action/Inaction
What if we don't solve this problem at this point?
What impact is there beyond the problem statement if we fix the problem now?
Further Information
Use this section for pertinent information and/or links to background information.
Suggested Reviewers
@mention individuals who you would like to have review this proposal. All are welcome to review and comment, but SMEs or others who should be aware of this EP should be listed here.
- [ ] @reviewer1 - [ ] @reviewer2
Comment Period
Specify a date by which this proposal's comment period should close. In order to ensure that lack of review does not delay mission critical enhancements, the suggested comment period is at least 7 calendar days, not including major holidays/breaks. Use your best judgment. If this is not time critical specify a longer comment period or none at all.
Comment period for this proposal shall close on [[July 28, 202X]].