krkn-chaos / krkn-hub

Containerized wrapper around https://github.com/krkn-chaos/krkn to inject failures into Kubernetes clusters with minimal configuration.
Apache License 2.0
18 stars 18 forks source link
chaos-engineering kubernetes openshift performance reliability resiliency

Krkn-hub aka kraken-hub

Hosts container images and wrapper for running scenarios supported by Krkn, a chaos testing tool for Kubernetes clusters to ensure it is resilient to failures. All we need to do is run the containers with the respective environment variables defined as supported by the scenarios without having to maintain and tweak files!

Supported chaos scenarios

Scenario Description Working
Pod failures Injects pod failures :heavy_check_mark:
Container failures Injects container failures based on the provided kill signal :heavy_check_mark:
Node failures Injects node failure through OpenShift/Kubernetes, cloud API's :heavy_check_mark:
zone outages Creates zone outage to observe the impact on the cluster, applications :heavy_check_mark:
time skew Skews the time and date :heavy_check_mark:
Node cpu hog Hogs CPU on the targeted nodes :heavy_check_mark:
Node memory hog Hogs memory on the targeted nodes :heavy_check_mark:
Node IO hog Hogs io on the targeted nodes :heavy_check_mark:
Service Disruption Deleting all objects within a namespace :heavy_check_mark:
Application outages Isolates application Ingress/Egress traffic to observe the impact on dependent applications and recovery/initialization timing :heavy_check_mark:
Power Outages Shuts down the cluster for the specified duration and turns it back on to check the cluster health :heavy_check_mark:
PVC disk fill Fills up a given PersistenVolumeClaim by creating a temp file on the PVC from a pod associated with it :heavy_check_mark:
Network Chaos Introduces network latency, packet loss, bandwidth restriction in the egress traffic of a Node's interface using tc and Netem :heavy_check_mark:
Pod Network Chaos Introduces network chaos at pod level :heavy_check_mark:
Service Hijacking Hijacks a service http traffic to simulate custom HTTP responses :heavy_check_mark:
SYN Flood Simulates a user-defined surge of TCP SYN requests directed at one or more services :heavy_check_mark:

Utilities

Utility Description Working
Chaos Recommender Runs the chaos recommender :heavy_check_mark:

Set Up

You can use docker or podman to run kraken-hub

Install Podman your certain operating system based on these instructions

or

Install Docker

Docker is also supported but all variables you want to set (separate from the defaults) need to be set at the command line In the form -e <VARIABLE>=<value>

You can take advantage of the get_docker_params.sh script to create your parameters string This will take all environment variables and put them in the form "-e =" to make a long string that can get passed to the command

For example: docker run $(./get_docker_params.sh) --net=host -v <path-to-kube-config>:/home/krkn/.kube/config:Z -d quay.io/redhat-chaos/krkn-hub:power-outages

TIP: Because the container runs with a non-root user, ensure the kube config is globally readable before mounting it in the container. You can achieve this with the following commands: kubectl config view --flatten > ~/kubeconfig && chmod 444 ~/kubeconfig && docker run $(./get_docker_params.sh) --name=<container_name> --net=host -v ~kubeconfig:/home/krkn/.kube/config:Z -d quay.io/krkn-chaos/krkn-hub:<scenario>

Adding New Scenarios/Testing Changes

Refer to the 2 docs below to be able to test your own images with any changes and be able to contribute them to the repository