Enhanced KCL test framework and tools to improve the code quality and stability, and help developers of the kcl repo or library gain more confidence after modifying their KCL code for example in the module repo: https://github.com/kcl-lang/modules.
The community basically divides "configuration validation and testing" into several phases:
Unit testing and compliance verification: with no real resources created/destroyed, the unit testing just verifies the expected behaviors of functions, logical code blocks, the input parameters of the program, and rules about the attribute values
Integration testing: real resources are created and destroyed. Usually in three steps:
Publish the configuration to runtime.
Check whether the resources created in runtime conform to expectations.
Clear test resources. Therefore, integration testing has a certain cost compared to unit testing
End-to-end testing: not only involves the operation of the real resources but also verifies the impact of the configuration on app services. Therefore, end-to-end testing makes great demands on the environment. Generally problems can be detected through monitoring
Basic lib developers and end users are distinguished in terms of scenarios of configuration test:
Basic lib developers: they are responsible for maintaining the basic models logic. Before releasing those libs, unit tests and integration tests need to be run. The integration test can be a three-step action of create-validate-destroy on the downstream configurations that depend on those libs. Therefore integration test costs more than unit test, and the high concurrency of the code change even exacerbate the cost. So the timing to run integration test is a trade off between time/resource cost and efficiency.
End users: they are usually app owners and sres. Before applying the code changes to the production, unit test, integration test, and end-to-end test need to be run. There are two big difference between the base and end test:
there can be some application-level validations that only apply to certain app configuration, and these validations are also need to be checked before merged to the main branch. So that end users could gain more confidence after updating application-level configs.
End-to-end testing is required for end users. To avoid actual impact on services in the production environment, end-to-end testing should be performed in the dev/test environment.
With the help of KCL language tools, the configuration verification in the local development - integration - end to end process is connected.
The verification and test of KCL code can be divided into three phases: local development, continuous integration, and end-to-end verification:
Cover the local and CI stage with kcl fmt, kcl lint, and kcl test tools to ensure code normalization and readability, improve configuration correctness and stability, and help detect configuration errors earlier.
Describe alternatives you've considered:
Provide kcl test command-line tool, integrate shortcut entry in KCL IDE. Integrate to internal/external CICD pipeline and Git hook.
Features about enhanced kcl test tool
Feature | Scenario(what it solves) | milestone
-- | -- | --
basic test capacity | Trigger tests with cli under specific directory, and during the execution output the real-time logging, and produce a summary of all the test cases; support the following test coding interface, including the given-when-then pattern, some helper function to improve readability |
tools and ecology engagement | tools: embed the kcl deps tools to support partial testing; support quick entrance in popular ide platform; support popular protocol and specs about test coverage and test report; embedded in github/gitlab action |
skipped test & specify which case to test | uses can input matching regex to execute specific tests when triggered by cli; Users can test a single case when triggered in IDE; the CI pipeline can trigger specific test cases according the kusion deps result |
coverage | after landing in the kcl repository, a test coverage threshold can be set to improve the code quality and ensure that the import models/schemas/attributes are under test. | support step by step: 1. first support test coverage on schema level and then detail to schema attribute level; 2. branch coverage and line coverage require the capacity of core KCL runtime.
test report | support test report specs such as xUnit, and engage to existing CICD tools to run quantitative analysis and visual presentation |
multi-input of one test case | Improve test code reuse |
mock | Two typical scenarios：1. kcl plugin enables access to external systems such as app meta center, cmdb, and those part of code should be mocked in ut. 2. the project_context plugin will check the directory of the entrance files and read the file content of project.yaml/stack.yaml, and that plugin is widely used in nearly all of the basic libs in konfig repo, and limits the flexibility to test basic libs since the files under test must be under a standard stack structure | the necessity and priority to be decided by users requirement
benchmark | The performance of the configuration code is quantifiable and can be used as input when other platforms wants to integrate | to be decided by users of both platform users and
fuzzing | can be used to test error-prone and complex logics and functions |
performance | parallel execution of test cases | first support pseudo concurrent, that is to say, just concurrently call the compile API concurrently
single-step debug | |
### Test code interface design
Test suite definition: Lambda function literal variable starting with `test_` located in the `*_test.k` file.
#### Code Test Example 1
+ main.k
```python
jsonpatch: (any, str) -> any = lambda ... # Omit the result
```
+ main_test.k
``` python
test_custom_log_volume_test = lambda {
data = {
a = [1, 2, 3]
}
assert jsonpatch(data, "a/0", 2).a[0] == 2, "2", "error message"
}
```
### Semantic level enhancement of kcl deps tool
The kcl deps tool need to be enhanced at the semantic level to identify which stacks have dependencies on a given basic schema/attribute
- Before execution of ```kcl test``` command, users need to know the scope to test, and that will be the input to specify the test cases.
- After modifying the logic in basic libs, lib developers need to know the impact on the downstream stacks: which stacks are affected, and the what changes happened to the final configuration data of those affected stacks, so that they can decide if the basic lib change could be released.
## User Interface and Workflow
```shell
kcl test [OPTIONS] [TESTNAME]
```
+ TESTNAME: If specified, only run tests containing this string in their names
```shell
kcl test # test the current package
kcl test test_jsonpatch # run tests containing this string in their names
```
## Impl Interface
+ TestRunner
```rust
pub struct TestResult {
pub errors: Vec,
}
trait TestRunner {
type Suite;
fn look_up_test_suites>(work_dir: P, package: &str) -> &[Self::Suite];
fn run(suites: &[Self::Suite]) -> Result<[TestResult]>;
}
```
+ Reporter
```rust
trait TestReporter {
fn report(suite: &TestResult, w: Writer) -> Result<()>;
}
```
## Teachability, Documentation, Adoption, Migration Strategy
Migration Strategy:
1. add test code on basic libs: https://github.com/kcl-lang/artifacthub
2. support kcl test in local and CI stages
## Reference
- google sre workbook - Configuration Specifics: https://sre.google/workbook/configuration-specifics/
- terraform test design: https://www.terraform.io/language/modules/testing-experiment
- terraform testing: https://www.hashicorp.com/blog/testing-hashicorp-terraform
- best practice on Terraform code from Azure: https://docs.microsoft.com/en-us/azure/developer/terraform/best-practices-testing-overview
- terraform compliance: https://terraform-compliance.com/
- terratest: https://terratest.gruntwork.io/
- jsonnet test: https://dev.to/cuotos/unit-testing-complex-jsonnet-objects-and-arrays-kpe
- opa test: https://www.openpolicyagent.org/docs/latest/policy-testing/
- cuelang test: https://github.com/KurtRudolph/cuelang-testing
app level compliance test: end users like app owners need to track the backend output and to check if the backend data meet the constraint in app's level. They've developed some postscripts to do those validations. It's better to be included in KCL unit test. for example, may look like this:
for unit tests on basic libs: treating all of the stacks as test cases can make basic lib developer feel safer before release, so it's meaningful and can be kept as it is for now. In the future, if basic lib testing causes too much time and has a significant efficiency effect, the concept of basic lib version management can be introduced to konfig repo. From the perspective of software development, involve version and releasing management to basic lib development. For example, one week or one day as a developing iteration and only trigger full stack test in each iteration.
Debugging capability and test coverage require a separate debugging toolchain and KCL Runtime support, and separate issues can be tracked and linked here.
konfig's CI pipeline can integrate Terratest to perform end to end testing.
Terratest has developed a bundle of go SDK with some helper functions and patterns to provide end-to-end test for popular runtime including terraform, k8s, packer, AWS, Azure, Google CloudPlatform, OpenPolicyAgent, docker.
The thought of terratest is mainly about using popular GPL like go, perform a set of "apply-validate-destroy" operation based on calling k8s/terraform/.. runtime API to check if a single application work in the real world or, multiple applications work together. And for checking if the application work as expected, it uses HTTP calls, accessing the cloud services' API, or other ways to connect to the service according to the runtime entity it bases on.
Feature Request about the KCL test tool
Describe the feature you'd like:
Enhanced KCL test framework and tools to improve the code quality and stability, and help developers of the kcl repo or library gain more confidence after modifying their KCL code for example in the module repo: https://github.com/kcl-lang/modules.
The community basically divides "configuration validation and testing" into several phases:
Unit testing and compliance verification: with no real resources created/destroyed, the unit testing just verifies the expected behaviors of functions, logical code blocks, the input parameters of the program, and rules about the attribute values
Integration testing: real resources are created and destroyed. Usually in three steps:
End-to-end testing: not only involves the operation of the real resources but also verifies the impact of the configuration on app services. Therefore, end-to-end testing makes great demands on the environment. Generally problems can be detected through monitoring
Basic lib developers and end users are distinguished in terms of scenarios of configuration test:
Basic lib developers: they are responsible for maintaining the basic models logic. Before releasing those libs, unit tests and integration tests need to be run. The integration test can be a three-step action of create-validate-destroy on the downstream configurations that depend on those libs. Therefore integration test costs more than unit test, and the high concurrency of the code change even exacerbate the cost. So the timing to run integration test is a trade off between time/resource cost and efficiency.
End users: they are usually app owners and sres. Before applying the code changes to the production, unit test, integration test, and end-to-end test need to be run. There are two big difference between the base and end test:
ref: https://sre.google/workbook/configuration-specifics/
With the help of KCL language tools, the configuration verification in the local development - integration - end to end process is connected.
The verification and test of KCL code can be divided into three phases: local development, continuous integration, and end-to-end verification:
Describe alternatives you've considered:
Provide kcl test command-line tool, integrate shortcut entry in KCL IDE. Integrate to internal/external CICD pipeline and Git hook.
Features about enhanced kcl test tool
from shupeng:
Debugging capability and test coverage require a separate debugging toolchain and KCL Runtime support, and separate issues can be tracked and linked here.
design on end-to-end testing
konfig's CI pipeline can integrate Terratest to perform end to end testing.
Terratest has developed a bundle of go SDK with some helper functions and patterns to provide end-to-end test for popular runtime including terraform, k8s, packer, AWS, Azure, Google CloudPlatform, OpenPolicyAgent, docker.
The thought of terratest is mainly about using popular GPL like go, perform a set of "apply-validate-destroy" operation based on calling k8s/terraform/.. runtime API to check if a single application work in the real world or, multiple applications work together. And for checking if the application work as expected, it uses HTTP calls, accessing the cloud services' API, or other ways to connect to the service according to the runtime entity it bases on.