Closed fanchibaoLiu closed 5 months ago
This is too old, and we do not provide support for personal blog posts. Please refer to https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/docs/install.md
This is too old, and we do not provide support for personal blog posts. Please refer to https://github.com/kubernetes-sigs/alibaba-cloud-csi-driver/blob/master/docs/install.md
think you,But if you uninstall the old version and then install the new version, will it have an impact on the previously created and mounted volumes?
If it has no impact on the data and does not affect the pod currently mounted on NAS, then I think this can be done.
Running Pods are usually not affected.
Your posted events seems indicating the csi-plugin is not running on your master node. Please check its log.
pod:
[root@kubernetes-master-001 ~]# kubectl get pod -o wide -n kube-system | grep csi
csi-plugin-5tvvd 2/2 Running 0 31d kubernetes-node-003 <none> <none>
csi-plugin-5xwnj 2/2 Running 0 31d kubernetes-node-002 <none> <none>
csi-plugin-jghj6 2/2 Running 0 22h kubernetes-master-001 <none> <none>
csi-plugin-qlkg6 2/2 Running 0 31d kubernetes-node-001 <none> <none>
csi-provisioner-65b5c6bc84-zk6bc 2/2 Running 0 30d kubernetes-master-001 <none> <none>
master csi logs:
[root@kubernetes-master-001 ~]# kubectl logs -f -n kube-system csi-plugin-jghj6 -c csi-plugin
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 12 100 12 0 0 824 0 --:--:-- --:--:-- --:--:-- 857
Running nas plugin....
time="2024-05-07T11:48:01+08:00" level=info msg="Multi CSI Driver Name: nas, nodeID: kubernetes-master-001, endPoints: unix://var/lib/kubelet/csi-plugins/driverplugin.csi.alibabacloud.com-replace/csi.sock"
time="2024-05-07T11:48:01+08:00" level=info msg="CSI Driver Branch: 'master', Version: 'v1.18.8.47-906bd535-aliyun', Build time: '2021-05-13-20:56:55'\n"
time="2024-05-07T11:48:01+08:00" level=info msg="Create Stroage Path: /var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/controller"
time="2024-05-07T11:48:01+08:00" level=info msg="Create Stroage Path: /var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/node"
time="2024-05-07T11:48:01+08:00" level=info msg="CSI is running status."
time="2024-05-07T11:48:01+08:00" level=info msg="Metric listening on address: /healthz"
time="2024-05-07T11:48:01+08:00" level=info msg="Driver: nasplugin.csi.alibabacloud.com version: 1.0.0"
I0507 11:48:01.862318 20214 driver.go:93] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0507 11:48:01.862327 20214 driver.go:81] Enabling controller service capability: CREATE_DELETE_VOLUME
I0507 11:48:01.862332 20214 driver.go:81] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0507 11:48:01.862336 20214 driver.go:81] Enabling controller service capability: EXPAND_VOLUME
W0507 11:48:01.862346 20214 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
time="2024-05-07T11:48:01+08:00" level=info msg="Metric listening on address: /metrics"
time="2024-05-07T11:48:01+08:00" level=info msg="Not found configmap named as csi-plugin under kube-system, with error: configmaps \"csi-plugin\" not found"
time="2024-05-07T11:48:01+08:00" level=info msg="Describe node kubernetes-master-001 and set RunTimeClass to runc"
time="2024-05-07T11:48:01+08:00" level=info msg="Node InternalIP is: 10.xxx.xxx.92"
time="2024-05-07T11:48:01+08:00" level=info msg="NAS Global Config: { false false false false runc kubernetes-master-001 10.xxx.xxx.92 false 0xc0004cc2c0 <nil>}"
time="2024-05-07T11:48:01+08:00" level=error msg="GetSTSToken: request roleInfo with error: parse \"http://100.100.100.200/latest/meta-data/ram/security-credentials/<?xml version=\\\"1.0\\\" encoding=\\\"iso-8859-1\\\"?>\\n<!DOCTYPE html PUBLIC \\\"-/W3C/DTD XHTML 1.0 Transitional/EN\\\"\\n \\\"http:/www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\\\">\\n<html xmlns=\\\"http:/www.w3.org/1999/xhtml\\\" xml:lang=\\\"en\\\" lang=\\\"en\\\">\\n <head>\\n <title>404 - Not Found</title>\\n </head>\\n <body>\\n <h1>404 - Not Found</h1>\\n </body>\\n</html>\\n\": net/url: invalid control character in URL"
time="2024-05-07T11:48:01+08:00" level=info msg="Get AK: use STS"
time="2024-05-07T11:48:01+08:00" level=info msg="NewControllerServer: current provisioenr nas limit is 2"
W0507 11:48:01.912569 20214 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0507 11:48:01.913267 20214 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock", Net:"unix"}
When creating a new mount volume, there are no new logs, only the pod event shows mount failure. At present, these logs are also have in other nodes.
When creating a new mount volume, there are no new logs, only the pod event shows mount failure.
That may indicate your kubelet is not using /var/lib/kubelet
as root dir. You may check the log of registrar container.
Yes, I have customized and specified the path for kubelet here: /data/kubelet/
The container to be mounted is always in the ContainerCreating state:
grafana-75f94d9d88-ndfp5 0/1 ContainerCreating 0 2m29s
# kubectl logs -f -n loki grafana-75f94d9d88-ndfp5
Error from server (BadRequest): container "grafana" in pod "grafana-75f94d9d88-ndfp5" is waiting to start: ContainerCreating
# describe
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 110s default-scheduler Successfully assigned loki/grafana-75f94d9d88-ndfp5 to kubernetes-master-001
Warning FailedMount 47s (x8 over 110s) kubelet MountVolume.MountDevice failed for volume "nas-1ca03f15-eadb-4908-b6a4-15460bb6480c" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name nasplugin.csi.alibabacloud.com not found in the list of registered CSI drivers
I confirmed again, master kubelet = /data/kubelet, node kubelet = /var/lib/kubelet, csi-plugin mount=/var/lib/kubelet.
This should not be reasonable,The problem should be right here.
Indeed, I have changed all the root directories of kubelet to /var/log/kubelet,
Mount again, successful! Thank you very much for your reminder.
After installing the CSI plugin in the k8s built by Kubeadm, the master node reported an error: failed to create newCsiDriverClient: driver name nasplugin.csi.alibabacloud.com not found in the list of registered CSI drivers
The node is not a problem and can be mounted normally.
Events: Warning FailedMount 46s (x58 over 102m) kubelet MountVolume.MountDevice failed for volume "nas-176f411d-a710-4954-b064-e3c2add7817e" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name nasplugin.csi.alibabacloud.com not found in the list of registered CSI drivers
`# kubectl get csinodes.storage.k8s.io
NAME DRIVERS AGE kubernetes-master-001 0 30d kubernetes-node-001 1 30d kubernetes-node-002 1 30d kubernetes-node-003 1 30d `
Environment:
kubectl version
): v1.28.2cat /etc/os-release
): CentOS7uname -a
): 4.19.12-1.el7.elrepo.x86_64Can you help me take a look at this issue? Are there any special restrictions on the master node?