This helm chart is designed to deploy functionality that automatically saves core dumps from most public cloud kubernetes service providers and private kubernetes instances to an S3 compatible storage service.
Please read the CONTRIBUTING.md it has some important notes. Pay specific attention to the Coding style guidelines and the Developer Certificate of Origin
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
The full code of conduct is available here
Please refer to the chart README.md for full details.
This is a matrix of confirmed test targets. Please PR environments that are also known to work
Provider | Product | Version | Validated? | Working? |
AWS | EKS | 1.21 | Yes | Yes |
AWS | ROSA | 4.8 | Yes | Yes |
Custom Build | K8S | N/A | Yes | Yes |
Digital Ocean | K8S | 1.21.5-do.0 | Yes | Yes |
GKE-cos_containerd | 1.20.10-gke.1600 | Yes | Yes | |
GKE-Ubuntu | 1.20.10-gke.1600 | Yes | Yes | |
IBM | IKS | 1.19-1.21 | Yes | Yes |
IBM | ROKS | 4.6-4.8 | Yes | Yes |
Microsoft | AKS | 1.19 | Yes | Yes |
Microsoft | ARO | 4.8 | Yes | Yes |
RedHat | On-Premises | 4.8 | Yes | Yes |
Core Dumps are a critical part of observability.
As systems become more distributed core dumps offer teams a non-invasive approach to understanding why programs are malfunctioning in any environment they are deployed to.
Core Dumps are useful in a wide number of scenarios but they are very relevant in the following cases:
The process exits without a useful stack trace
The process runs out of memory
An application doesn’t behave as expected
The traditional problems with core dumps are:
Overhead of managing the dumps
Dump Analysis required specific tooling that wasn't readily available on the developers machine.
Managing Access to the dumps as they can contain sensitive information.
This chart aims to tackle the problems surrounding core dumps by leveraging common platforms (K8s, ROKS and Object Storage) in a cloud environment to pick up the heavy lifting.
The chart deploys two processes:
The agent manages the updating of /proc/sys/kernel/*
configuration, deploys the composer service and uploads the core dumps zipfile created by the composer to an object storage instance.
The composer handles the processing of a core dump and creating runtime, container coredump and image JSON documents from CRICTL and inserting them into a single zip file. The zip file is stored on the local file system of the node for the agent to upload.
When you install the IBM Cloud Core Dump Handler Helm chart, the following Kubernetes resources are deployed into your Kubernetes cluster:
Namespace: A specific namespace is created to install the components into - defaults to ibm-observe
Handler Daemonset: The daemonset deploys a pod on every worker node in your cluster. The daemonset contains configuration to enable the elevated process to define the core pattern to place the core dump into object storage as well as gather pod information if available.
Privileged Policy: The daemonset configures the host node so priviledges are required.
Service Account: Standard Service account to run the daemonset
Volume Claims: For copying the composer to the host and enabling access to the generated core dumps
Cluster Role: Created with an event resource and create verb and associated with the service account.
To install the Helm chart in your cluster, you must have the Administrator platform role.
This chart deploys privileged kubernetes daemonset with the following implications:
the automatic creation of privileged container per kubernetes node capable of reading core files querying the crictl for pod info.
The daemonset uses hostpath feature interacting with the underlying Linux OS.
The composer binary is deployed and ran on the host server
Core dumps can contain sensitive runtime data and the storage bucket access must be managed accordingly.
Object storage keys are stored as secrets and used as environment variables in the daemonset
The IBM Cloud Core Dump Handler requires the following resources on each worker node to run successfully:
$ helm delete core-dump-handler --namespace observe
host-name
are deleted before continuing
$ kubectl get pvc -n observe
$ helm install core-dump-handler . --namespace observe
helm delete core-dump-handler -n observe
Build the image docker build -t YOUR_TAG_NAME .
Push the image to your container registry
Update the container in the values.yaml
file to use it.
image:
registry: YOUR_REGISTRY
repository: YOUR_REPOSITORY
tag: YOUR_TAG
or run the helm install command with the settings
--set image.registry=YOUR_REGISTRY \
--set image.repository=YOUR_REPOSITORY \
--set image.tag=YOUR_TAG
The services are written in Rust using rustup.
Local unit tests can be ran using cargo test
in the base folder
Currently only IBM Cloud ROKS and IKS are supported but we are happy to carry integration tests for other services but we can't run them before release.
To run the integration tests build follow the instructions for a custom build
In the root of the project folder create a file called .env
with the following configuration
S3_ACCESS_KEY=XXXX
S3_SECRET=XXXX
S3_BUCKET_NAME=XXXX
S3_REGION=XXXX
Change directory to the integration folder and run the test
cd integration
./run-ibm.sh
Releases are built on a pre-release branch e.g. pre-8.5.0
integration tests are ran manually and a release is generated when merged to main.
It currently isn't possible to automate this as the kubernetes integration in github actions is not reliable enough.
If you wish to test a pre-release with your own integration testing then please raise an issue and we can collaborate on your test run.
The first place to look for issues is in the agent console. A successful install should look like this
[2021-09-08T22:28:43Z INFO core_dump_agent] Setting host location to: /var/mnt/core-dump-handler
[2021-09-08T22:28:43Z INFO core_dump_agent] Current Directory for setup is /app
[2021-09-08T22:28:43Z INFO core_dump_agent] Copying the composer from ./vendor/default/cdc to /var/mnt/core-dump-handler/cdc
[2021-09-08T22:28:43Z INFO core_dump_agent] Starting sysctl for kernel.core_pattern /var/mnt/core-dump-handler/core_pattern.bak
[2021-09-08T22:28:43Z INFO core_dump_agent] Created Backup of /var/mnt/core-dump-handler/core_pattern.bak
[2021-09-08T22:28:43Z INFO core_dump_agent] Starting sysctl for kernel.core_pipe_limit /var/mnt/core-dump-handler/core_pipe_limit.bak
[2021-09-08T22:28:43Z INFO core_dump_agent] Created Backup of /var/mnt/core-dump-handler/core_pipe_limit.bak
[2021-09-08T22:28:43Z INFO core_dump_agent] Starting sysctl for fs.suid_dumpable /var/mnt/core-dump-handler/suid_dumpable.bak
[2021-09-08T22:28:43Z INFO core_dump_agent] Created Backup of /var/mnt/core-dump-handler/suid_dumpable.bak
[2021-09-08T22:28:43Z INFO core_dump_agent] Created sysctl of kernel.core_pattern=|/var/mnt/core-dump-handler/cdc -c=%c -e=%e -p=%p -s=%s -t=%t -d=/var/mnt/core-dump-handler/core -h=%h -E=%E
kernel.core_pattern = |/var/mnt/core-dump-handler/cdc -c=%c -e=%e -p=%p -s=%s -t=%t -d=/var/mnt/core-dump-handler/core -h=%h -E=%E
kernel.core_pipe_limit = 128
[2021-09-08T22:28:43Z INFO core_dump_agent] Created sysctl of kernel.core_pipe_limit=128
fs.suid_dumpable = 2
[2021-09-08T22:28:43Z INFO core_dump_agent] Created sysctl of fs.suid_dumpable=2
[2021-09-08T22:28:43Z INFO core_dump_agent] Creating /var/mnt/core-dump-handler/.env file with LOG_LEVEL=info
[2021-09-08T22:28:43Z INFO core_dump_agent] Executing Agent with location : /var/mnt/core-dump-handler/core
[2021-09-08T22:28:43Z INFO core_dump_agent] Dir Content []
If the agent is running successfully then there may be a problem with the composer configuration. To check the logs for the composer open a shell into the agent and cat the composer.log to see if there are any error messages.
cat /var/mnt/core-dump-handler/composer.log
If there are no errors then you should change the default log from error
to debug
in the values.yaml and redeploy the chart.
Create a core dump again and /var/mnt/core-dump-handler/composer.log
should contain specific detail on each upload.