lubronzhan
commented
1 year ago Thanks for opening up. About this issue, when initializing the node, the Cloud-provider-vsphere will first try find the providerID of each node by searching with node name. Related code. Then the Cloud provider vSphere will try find VM by searching by DNS Name. And only return the first result from the list
I noticed the node name you use in the cluster are form like master0 master1, I wonder are they within the same vSphere, if that's the case, then it could be a case that CPI find different node with same (node name)/(dns name) in the same vsphere.
In CAPV we use distinct node name for each cluster node, each node also has distinct DNS name, so each cluster won't collide with same node name. I wonder how could different VM has same name in Openshift cluster in vSphere? Or they are using different VM name but same DNS name? Could you share those VM information? Does nodes in different clusters has same DNS name? You are using single VC right?
The log is missing some of the initialization parts, so I can't verify that the duplicate DNS Name is the root cause. Could you create a new cluster and install CPI with log level setting to 5, then return the log to me？ I was expecting to see log in below code
https://github.com/kubernetes/cloud-provider-vsphere/blob/master/pkg/cloudprovider/vsphere/instances.go#L98 https://github.com/kubernetes/cloud-provider/blob/master/controllers/node/node_controller.go#L415
So we could see during the initialization, CPI is returning the wrong provider ID for corresponding node name or not.
parogui
k8s-triage-robot
k8s-ci-robot
What happened?
After installing CSI driver, we noticed CPI is discovering nodes out of the cluster (OpenShift in this case) VMs folder.
If the nodes are manually deleted, they are re-discovered.
The only way we were able to workaround the issue is by reducing the permissions on the user used for vSphere so it only has access to its cluster, making us require a per-cluster vSphere user with permissions to their own folder.
What did you expect to happen?
We'd expect the CPI discovery to hone the folder configuration on the vSphere CSI driver configmap to prevent discovering nodes that are not supposed to be present in node discovery.
How can we reproduce it (as minimally and precisely as possible)?
Install vSphere CSI driver on OpenShift 4.x on a vSphere cluster where more than one vSphere cluster co-exists on different folders.
Anything else we need to know (please consider providing level 4 or above logs of CPI)?
Every 5 minutes node discovery is run and adds the nodes that were removed because they do not belong to the cluster.
Kubernetes version
Cloud provider or hardware configuration
OS version
Kernel (e.g.
uname -a)Install tools
Container runtime (CRI) and and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)
Others