Expand test scenarios

[bakhtin]

Each Helm chart is now configured to run a smoke test on each PR. A chart is deployed to a local K8s cluster with default or close-to-default parameters. While smoke tests are still good to capture basic errors, they don't cover real-world usage scenarios.

To prevent the bugs from ending up in the release more comprehensive end-to-end tests should be added to the CI pipeline. But these tests should not repeat or serve as a replacement for the tests of the application itself. Rather the tests should capture Helm chart specific errors like improperly exposed ports, failures to attach PVCs, bugs in the init-scripts, etc.

The values.yaml file should also be extended with more comprehensive configuration reflecting how the chart is used in the wild. Potentially, having multiple values.yamls to cover different use-cases.

[bakhtin]

Check out https://github.com/bitnami/charts/blob/main/TESTING.md

[bakhtin]

As the first step I'll pick up node Helm chart to implement tests for it. Node Helm chart is the most used and complex chart in the repo.

The initial iteration should cover the most used configuration options and the pieces of functionality that has the most impact on the runtime. I came up with the following:

Test the following node modes:

• Validator
• RPC
• Full
• Bootnode

Test scenarios:

• a node is running in --dev mode along side with `ws-exporter`` sidecar and posting a healthy status (relay chain + para chain)
• backup restoration works. Need to mock it somehow, restoring full backup is not feasible. Possibly, serve a small backup for a custom chain from the local HTTP server spun up within the same test suite
• RPC node (2 replicas) is publicly exposed through the LB and serves RPC queries
• a Bootnode can bootstrap other nodes on the network
• a Validator nodes produce blocks (possible with --dev flag?)
• can load keys from Vault
• Pods are reachable through Services (p2p, rpc)
• can download custom chainspec
• can download custom runtime
• generated node key are persisted
• can inject keys

I found the Bitnami's testing guide comprehensive covering the details of possible strategies and tools for testing Helm charts. Bitnami's Helm charts repo uses 3 tools to run the tests:

• Cypress for web testing. Mostly useful to test the apps accessed using a web-browser. Tests are running using a headless browser. Less useful for API testing. substrate-telemetry and testnet-manager charts will benefit from it as they have a frontend component.
• GOSS for integration tests. Implements Serverspec. Validates certain properties are met (e.g., a file exists). Runs inside the container.
• Ginkgo also for integration tests. Useful for validating certain properties of the objects inside the K8s cluster (e.g., all replicas are ready or a PVC exists).

Each test scenario may require one or more tools to verify the requirements are met.

[bakhtin]

Covered a few scenarios in #304:

• [x] a node is running in --dev mode along side with ws-exporter sidecar and posting a healthy status (relay chain + para chain)
• [x] backup restoration works. Need to mock it somehow, restoring full backup is not feasible. Possibly, serve a small backup for a custom chain from the local HTTP server spun up within the same test suite
• [x] RPC node (2 replicas) is publicly exposed through the LB and serves RPC queries
• [ ] a Bootnode can bootstrap other nodes on the network
• [x] a Validator nodes produce blocks (possible with --dev flag?)
• [ ] can load keys from Vault
• [x] Pods are reachable through Services (p2p, rpc)
• [x] can download custom chainspec
• [ ] can download custom runtime
• [ ] generated node key are persisted
• [ ] can inject keys