Open scole-scea opened 6 months ago
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle stale
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/stale
is appliedlifecycle/stale
was applied, lifecycle/rotten
is appliedlifecycle/rotten
was applied, the issue is closedYou can:
/remove-lifecycle rotten
/close
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
/remove-lifecycle rotten
@scole-scea I had also faced this problem but overcome up with using consistent labelling of the nodes. Ensure that the kubernetes.io/hostname label matches the name of the node by setting it on each node correctly. We can achieve this using an init container to set the label.
Here are the Daemonset.yaml under
spec:
initContainers:
- name: set-node-label
image: busybox
command:
- sh
- -c
- |
NODE_NAME=$(cat /etc/hostname)
kubectl label node $NODE_NAME kubernetes.io/hostname=$NODE_NAME --overwrite
Hope it may help but yes the problem is valid as it delete the valid PVs.
Unfortunately, that's not a great answer for our environment. Hostnames are set by legacy rules; k8s node names have always been different and now are expected, etc, etc.
I'd love it if someone who knew this code could make the "easy change" (if such a thing exists); else I'll get to it maybe sometime this year.
I do appreciate your comments, though. It means my guesses about what the problem is are correct, which is a lovely confirmation.
Thank you for your feedback. I provided the most probable solution but yes, I can understand that our environment have specific requirement. We should look into it.
What happened:
In a cluster where the node name != kubernetes.io/hostname, the checker for node deletion often (but not always) thinks new PVs belong to deleted nodes, and deletes them. (Then the provisioner reprovisions them a few seconds later, causing constant churn.) Thankfully this only happens to unused volumes.
What you expected to happen:
I expect provisioned, valid PVs to be ignored by the deleter, even if they're not used.
How to reproduce it:
In a cluster where node name != kubernetes.io/hostname, launch the storage provisioner and add the node-cleanup service. Notice churn in the volumes.
Anything else we need to know?:
Environment:
v2.6.0
kubectl version
):v1.24.17-eks-5e0fdde
uname -a
):5.10.201-191.748.amzn2.x86_64