Closed bdbrink closed 1 year ago
Hi @bdbrink Thanks for taking the time to look at the project. I think you are on the right track. A failing cdc with no composer.log is likely caused by the wrong cdc being deployed. Unfortunately it is very hard to capture the libc incompatibility problem.
Can you confirm the --values flags used for the install?
As per the docs I would expect you used --values values.aws.yaml
I have looked at the v8.10.0 image and have confirmed the cdc is there
podman run -it quay.io/icdh/core-dump-handler:v8.10.0 /bin/bash
[root@67269174863d app]# ls ./vendor/rhel7/
cdc
So I can only assume that the copy failure on the second attempt is due to some issues mounting the volume after the reinstall. Can you uninstall the helm chart again and confirm the volumes have also been deleted before reinstalling with the values flag.
If the cdc still doesn't copy over can you log into the agent and try the copy manually?
cp ./vendor/rhel7/cdc /var/mnt/core-dump-handler/cdc
It seem like we have two possible issues here so apologies for all the questions.
@No9 Thanks for quick response!
I specified values.aws.sts.yaml
for the chart values, deleted the volumes and gave it a fresh redeploy but the error is still persisting, when specifying to default the agent begins to run, but still no composer.log
. The pods are crashlooping so I can't get into the agent to copy over manually (unless there is another way to do that). I'm using the arm64 image and the k8s nodes are using Amazon Linux2 with containerd for the runtime if any of that helps.
Can I clarify exactly where we are as I've got a feeling I'm going to have to try and replicate this:
You are running EKS 1.23 on arm64 with Linux2 worker nodes.
You are using the arm images from quay.io/icdh/core-dump-handler-musl:v8.10.0
so you have updated the main values.yaml
to be:
image:
registry: quay.io
repository: icdh/core-dump-handler-musl
tag: v8.10.0
or you are supplying --set image.repository=quay.io/icdh/core-dump-handler-musl
as an cli option.
When you run this values.aws.sts.yaml
with no other yaml changes or command options you get a crash loopback off.
# AWS requires a crio client to be copied to the server
daemonset:
includeCrioExe: true
vendor: rhel7 # EKS EC2 images have an old libc=2.26
serviceAccount:
annotations:
# See https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
eks.amazonaws.com/role-arn: arn:aws:iam::123456789000:role/iam-role-name-here
When you run it with default it works but still no composer log.
# AWS requires a crio client to be copied to the server
daemonset:
includeCrioExe: true
vendor: default # <---------- This is the change
serviceAccount:
annotations:
# See https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html
eks.amazonaws.com/role-arn: arn:aws:iam::123456789000:role/iam-role-name-here
Setting the composer log to info with --set composer.logLevel=Info
still doesn't provide any additional information after a core dump has been generated. Or in values.aws.sts.yaml
composer:
logLevel: "Info"
Please correct me if I have misunderstood something as I want to be sure I'm on the same config as you.
Yes thats the exact setup I'm using/running, doing this all in the helm charts (deploying with v3.9.4 of helm), not using the cli for any inputs.
OK so it looks like default
should be used as that successfully runs the agent.
I'll try and login to that and copy over the cdc
Then I will be trying to do is get onto a host node and run the cdc --help
to see if there are any dependency weirdness that has been introduced. Won't be for a few days though
Hello,
Using
v8.10.0
with EKS 1.23 with IRSA I was able to install the agent and get it running, but when viewing the/var/mnt/core-dump-handler
the composer.log is missing. When triggering a core dump nothing happens within the agent (assuming thats due to the missing log file).Agent running:
Executing into pod:
I've tried to update the vendor to
rhel7
according to other issues seen with EKS, but this throws an error: