The Virtual Kubelet Test Set provides a simple templated infrastructure to collect, run and report functionality tests for Virtual Kubelet providers. It is being developed in the context of the Proof-of-Concept on the adoption of the InterLink abstractions to serverless containers to ease the validation of the various backends.
InterLink is a toolset to provide an abstraction for the execution of a Kubernetes pod on any remote resource capable of managing a Container execution lifecycle.
It is composed of a client translating the kube-apiserver requests into InterLink APIs and forwarding them towards a remote target, that translate them back to interact with some container lifecycle management tool. To decouple the management of the container lifecycle from the APIs, the former is designed to be implemented in an independent container plugin, often called the sidecar in the InterLink jargon.
The Virtual Kubelet Test Set (vk-test-set) repository aims at providing a standard set
of tools to validate the functionality of the sidecar, by submitting Pods
of increasingly complexity. It features:
pytest;jinja2;pydantic.Templates are stored in vktestset/templates and shipped with the
Python package.
Each template is composed of three sections:
#
# The template documentation
#
The Kubernetes manifests
# VALIDATION
The validation configuration
Note in particular, that the line # VALIDATION separates the Kubernetes
manifests from the validation.
Below is reported the template for a hello world pod.
## Hello world
##
## First simple test to ensure the communication is open and a simple
## echo command can be executed.
## Launches a pod printing "hello world" in the log.
apiVersion: v1
kind: Pod
metadata:
name: hello-world-{{ uuid }}
namespace: {{ namespace }}
annotations: {{ annotations | tojson }}
spec:
restartPolicy: Never
nodeSelector:
kubernetes.io/hostname: {{ target_node }}
containers:
- name: hello-world
image: ghcr.io/grycap/cowsay
command: ["/bin/bash", "-c"]
args: ["echo hello world"]
imagePullPolicy: Always
dnsPolicy: ClusterFirst
tolerations: {{ tolerations | tojson }}
################################################################################
# VALIDATION
timeout_seconds: 10
check_pods:
- name: hello-world-{{ uuid }}
namespace: {{ namespace }}
check_logs:
- name: hello-world-{{ uuid }}
namespace: {{ namespace }}
container: hello-world
regex: hello world
operator: Exists
clean_configs:
- type: pod
name: hello-world-{{ uuid }}
namespace: {{ namespace }}
condition: always
The templates are configured through a YAML file, named by
default vktest_config.yaml.
The fields of the configuration file are:
target_nodes (list of strings), the Kubernetes names of the target nodes to be testedrequired_namespaces (optional list of strings), a list of namespace that should exist before other tests are runtimeout_multiplier (optional float), a multiplier for all the timeouts of the test.
It can be increased for unattended testing or lowered during debug with cached images on the remote target.values (dictionary) a dictionary of values passed as-is to the jinja2 templateAn example of configuration is reported below.
target_nodes:
- vk-hub-test-agent-1
required_namespaces:
- default
- kube-system
- interlink4
timeout_multiplier: 1.
values:
namespace: interlink4
annotations:
slurm-job.vk.io/flags: "--job-name=test-pod-cfg -t 2800 --ntasks=8 --nodes=1 --mem-per-cpu=2000"
tolerations:
- key: virtual-node.interlink/no-schedule
operator: Exists
effect: NoScheduleAn implicit variable passed to the jinja template together with the values dictionary
is uuid which is a random string used to ensure uniquness in
the names of the created resources.
The validation section is composed of three dictionaries:
check_pods defines checks on the status of a pod;check_logs defines checks based on regular expressions used to process the log
of a specified container in a given pod;clean_configs defines the clean-up operations to avoid cluttering the cluster
with completed (or failed) test pods;In addition, a timeout can be defined specifying the timeout_seconds variable.
The tests that wait for some condition to verify to succeed, will not wait longer
than the specified timeout.
See the auto-generated documentation in ValidationProcedure.md.
KUBECONFIG points to the configuration file of the connection to the kubernetes
cluster (as for kubectl). If empty,
or unset, in-cluster configuration is assumed. VKTEST_CONFIG points to the configuration file of the tests, defaults to
vktest_config.yaml.Make sure you have an active kubectl connection to the cluster where the virtual node is configured,
kubectl get nodesshould list the nodes defined in the cluster, including the virtual node.
Clone this repository
git clone git@github.com:landerlini/vk-test-set.git
cd vk-test-setConfigure the tests for your deployment
vim vktest_config.yamlRun all the tests, with enhanced verbosity (-v) and stopping on failure (-x).
pytest -vxRerun a single test (for example upon failure in the debugging loop) you can copy the string identifying the test in the pytest output as an argument of pytest itself.
For example,
pytest -v vktestset/basic_test.py::test_manifest[vk-hub-test-agent-1-070-rclone-bind.yaml]Please note that in the development of tests, features validated by former tests are usually assumed, so that related failures might result obfuscated and less obvious to identify.
In general, it is a bad idea to test advanced features if basic features are properly working.
Adding new tests is highly appreciated! We encourage contributions both on adding smaller tests that provide early validation of common pitfalls in the definition of a container plugin and more complex tests testing more advanced features.
In general, if you find a bug in a plugin escaping this test set, we want to know!