adejanovski
commented
6 months ago It looks like the seed service isn't pointing to any of the pre-existing nodes in the cluster, creating some sort of split brain situation. As the nodes are unable to connect with each other, they're stuck trying to communicate with the old IPs that are stores in the system tables.
These events are the problematic ones:
46m Warning FailedToUpdateEndpoint endpoints/k8ssandra-dc2-service Failed to update endpoint k8ssandra-operator/k8ssandra-dc2-service: Operation cannot be fulfilled on endpoints "k8ssandra-dc2-service": the object has been modified; please apply your changes to the latest version and try again
46m Warning FailedToUpdateEndpoint endpoints/k8ssandra-seed-serviceI'm not sure what generated that situation. cass-operator is responsible for placing the seed-node labels that the seed service uses to build the seed list. Did you try to move all 3 nodes at the same time, or one by one? Are there errors in the cass-operator container logs that could help us understand what's happening here? Could you also list the pods with their labels?
networkingana
burmanm
What did you do? I have deploymnet of k8ssandra over 5 kubernetes nodes, 3 of them were on CentOS 7 and 2 of them were on Ubuntu 20.04. The plan was to migrate all of the k8ssandra nodes from the CentOS 7 to new Kubernetes nodes on Ubuntu 20.04. The persistent volume of cassandra is created using the local-path-provisioner from rancher. Because my Kubernetes nodes are virtual machine, I moved the hdd device from one VM to another on the virtualization infrastructure, then rescheduled the pod of K8ssandra on that node, the same steps for all of the nodes.
I changed the node affinity in the PV to match the new node, and changed the PVC label to match the new node the same. The PVC were sucessfully bound but the Pods are in the CrashLoopBackOff state. I can see that URGENT MESSAGES on port 7000 are send on wrong IP address.
I tried restarting the k8ssandra cluster using the stopped: true and than stopped: false parameter in the k8ssandraclusters.k8ssandra.io CRD, but the issue is the same.
kubectl get events -n k8ssandra-operator
Here is the repeating log
Here are the pods with wide output, here we can see that the IP address from the above log is not found on the actual pods
Environment Kubernetes version: v1.22.3 OS: Ubuntu 20.04.2 LTS Docker version: docker://20.10.24
K8ssandra Operator version:
docker.io/k8ssandra/k8ssandra-operator:v1.2.1Kubernetes version information:
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.0", GitCommit:"c2b5237ccd9c0f1d600d3072634ca66cefdf272f", GitTreeState:"clean", BuildDate:"2021-08-04T17:57:25Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}Kubernetes cluster kind:
kubeadm
Manifests:
Please let me know if I can provide additional information. Thank you
┆Issue is synchronized with this Jira Story by Unito