Closed iameli closed 1 year ago
Thanks for the report @iameli - This is a tricky one alright
Can you provide logs from the agent container and the composer log in /home/data/core-dump-handler/composer.log
?
It would be good to confirm if it's the composer on the host or the agent in the container that's causing the issue.
From 10000 feet it seems like /home/data/core-dump-handler/cores
isn't getting mounted on provision but on first use or it's causing issues when accessed from the host and and this is leading to a race condition of some sort.
You may also want to see if you can create a file in the core folder outside of the core dump process with touch /home/data/core-dump-handler/cores/RANDOM_NAME
on the host to see if that forces a mount.
As an aside can you also confirm that a single core dump is working?
kubectl run -it segfaulter --image=quay.io/icdh/segfaulter --restart=Never
kubectl delete pod segfaulter
@iameli This has been open for a month with no further feedback so I'm closing it off as I think the issue is with the local-path-provisioner. If I'm wrong or you have more information please feel free to re-open this and I have also subscribed to the rancher/local-path-provisioner issue to make sure I get updates.
@No9 it looks like it's an easy issue to simulate, and it might be related to the Bidirectional
mode the spec use it.
Look this on the docs https://kubernetes.io/docs/concepts/storage/volumes/#local
Warning:
Bidirectional
mount propagation can be dangerous. It can damage the host operating system and therefore it is allowed only in privileged containers. Familiarity with Linux kernel behavior is strongly recommended. In addition, any volume mounts created by containers in pods must be destroyed (unmounted) by the containers on termination.
On every new corefile generated , it could find a new mount on the underlying machine
we are facing the same issue with thousands of mounts for ...cores being created. Is Bidirectional mode required?
it seems like every restart of the coredump handler causes the mounts to double. restarting the host resets this
I have no idea who to report this bug to, so I'm going to duplicate the report a few places. local-path-provisioner: https://github.com/rancher/local-path-provisioner/issues/287 kubernetes: https://github.com/kubernetes/kubernetes/issues/114583
Environmental Info: core-dump-handler image:
quay.io/icdh/core-dump-handler:v8.2.0
K3s Version:I have also seen this behavior on a different node running a more recent version
Node(s) CPU architecture, OS, and Version:
Cluster Configuration: 6 Server Nodes
Describe the bug: I'm running core-dump-handler on a few nodes. When core-dump-handler comes under load — we had a service elsewhere that was malfunctioning and segfaulting many times per second — its directory gets bind-mounted over and over and over and over. I do not know by whom.
Steps To Reproduce: No idea how to reproduce this in an isolated environment, but I'll give it a shot as I continue debugging.
Here's core-dump-handler's DaemonSet configuration file and the PVCs that back it. The pertinent volumes section:
Possibly a problem with bind-mounting one directory inside another...?