search

seaweedfs / seaweedfs-csi-driver

SeaweedFS CSI Driver https://github.com/seaweedfs/seaweedfs
Apache License 2.0
211 stars 50 forks source link

Cant create pod in PVC #40

Open sejesnet opened 2 years ago

sejesnet commented 2 years ago

Good morning. Having trouble with creating pods which have volume claime to seaweedfs-storage.

I reproduced same issues on Kubernetes cluster on 3 nodes and in minikube all latest versions.

Sorry for much text, i'm newbue in kubernetes. I show all steps, that I do for the same results all time :)

[root@kub1 seaweedfs-csi-driver-1.0.6]# helm install --set seaweedfsFiler=172.16.32.195:9337 seaweedfs-csi-driver ./deploy/helm/seaweedfs-csi-driver
NAME: seaweedfs-csi-driver
LAST DEPLOYED: Fri Oct 15 14:37:44 2021
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None

[root@kub1 seaweedfs-csi-driver-1.0.6]# k get storageclasses.storage.k8s.io 
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  8d
seaweedfs-storage      seaweedfs-csi-driver    Delete          Immediate              false                  148m

[root@kub1 seaweedfs-csi-driver-1.0.6]# k apply -f deploy/kubernetes/sample-seaweedfs-pvc.yaml
persistentvolumeclaim/seaweedfs-csi-pvc created

[root@kub1 seaweedfs-csi-driver-1.0.6]# k get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
fileshare           Bound    my-nfs-share                               10Gi       RWX                                77m
seaweedfs-csi-pvc   Bound    pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e   5Gi        RWO            seaweedfs-storage   3s

[root@kub1 seaweedfs-csi-driver-1.0.6]# k get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                       STORAGECLASS        REASON   AGE
my-nfs-share                               10Gi       RWX            Retain           Bound    default/fileshare                                        77m
pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e   5Gi        RWO            Delete           Bound    default/seaweedfs-csi-pvc   seaweedfs-storage            8s

[root@kub1 seaweedfs-csi-driver-1.0.6]# k apply -f deploy/kubernetes/sample-busybox-pod.yaml
pod/my-csi-app created

It sleep in ContainerCreating status infinetly:

[root@kub1 seaweedfs-csi-driver-1.0.6]# k get pods
NAME                                READY   STATUS              RESTARTS      AGE
my-csi-app                          0/1     ContainerCreating   0             12s

Simpe describe for pod, at the end you see Unable to attach or mount volumes:

[root@kub1 seaweedfs-csi-driver-1.0.6]# k describe pod my-csi-app
.................................
Volumes:
  seaweedfs-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  seaweedfs-csi-pvc
    ReadOnly:   false
  kube-api-access-9cw5h:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason              Age    From                     Message
  ----     ------              ----   ----                     -------
  Normal   Scheduled           2m26s  default-scheduler        Successfully assigned default/my-csi-app to kub3.snet
  Warning  FailedAttachVolume  27s    attachdetach-controller  AttachVolume.Attach failed for volume "pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e" : Attach timeout for volume pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e
  Warning  FailedMount         24s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[seaweedfs-volume], unattached volumes=[seaweedfs-volume kube-api-access-9cw5h]: timed out waiting for the condition

Logs from seaweedfs-filer (from logs you see that pvc volume was created in /bucket dir):

Oct 15 14:38:24 node2 seaweedfs-filer: I1015 14:38:24  1568 filer_grpc_server.go:139] CreateEntry /buckets/pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e
Oct 15 14:38:24 node2 seaweedfs-filer: I1015 14:38:24  1568 filer.go:196] find uncached directory: /buckets
Oct 15 14:38:24 node2 seaweedfs-filer: I1015 14:38:24  1568 filer.go:159] InsertEntry /buckets/pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e: new entry: pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e
Oct 15 14:38:24 node2 seaweedfs-filer: I1015 14:38:24  1568 filerstore_wrapper.go:128] InsertEntry /buckets/pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e
Oct 15 14:38:24 node2 seaweedfs-filer: I1015 14:38:24  1568 filer.go:181] CreateEntry /buckets/pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e: created
Oct 15 14:38:29 node2 seaweedfs-filer: I1015 14:38:29  1568 filer.go:196] find uncached directory: /topics/.system/log/2021-10-15
Oct 15 14:38:29 node2 seaweedfs-filer: I1015 14:38:29  1568 filer.go:159] InsertEntry /topics/.system/log/2021-10-15/11-38.7824ce35: new entry: 11-38.7824ce35
Oct 15 14:38:29 node2 seaweedfs-filer: I1015 14:38:29  1568 filerstore_wrapper.go:128] InsertEntry /topics/.system/log/2021-10-15/11-38.7824ce35
Oct 15 14:38:29 node2 seaweedfs-filer: I1015 14:38:29  1568 filer.go:181] CreateEntry /topics/.system/log/2021-10-15/11-38.7824ce35: created

Logs from kuber node, where pod deployed:

Oct 15 14:38:49 kub3 kubelet: E1015 14:38:49.894876    1050 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/seaweedfs-csi-driver^pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e podName: nodeName:}" failed. No retries permitted until 2021-10-15 14:38:57.894854658 +0300 MSK m=+48959.413196029 (durationBeforeRetry 8s). Error: Volume not attached according to node status for volume "pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e" (UniqueName: "kubernetes.io/csi/seaweedfs-csi-driver^pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e") pod "my-csi-app" (UID: "fb8599a5-a1d9-4dd4-a140-39b83b7eea6c")
Oct 15 14:38:57 kub3 kubelet: I1015 14:38:57.944563    1050 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e\" (UniqueName: \"kubernetes.io/csi/seaweedfs-csi-driver^pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e\") pod \"my-csi-app\" (UID: \"fb8599a5-a1d9-4dd4-a140-39b83b7eea6c\") "

If you ask for weed binary and fuse packegs, I installed it to all cluster nodes, and make symlink for 

ln -s /usr/sbin/weed /usr/sbin/mount.weed

I can mount bucket by typing:

mount -t weed fuse /some_dir -o "filer='filler_ip_addres',filer.path=/datadir"

and I see here directory of bucket where placed empty pvc-84e08db8-2c99-42af-b72a-e1911a67ad1e dir.

Could you please help me solve problem deploying pods on seaweedfs volums thought seaweedfs-csi-driver. May be I something miss.

xiaoweihong commented 2 years ago

I have same error

xiaoweihong commented 2 years ago 
root@k8s-master-27:~/seaweedfs-csi-driver-1.0.9# kubectl describe pods my-csi-app 
Name:         my-csi-app
Namespace:    default
Priority:     0
Node:         k8s-master-27/192.168.2.27
Start Time:   Tue, 10 May 2022 16:09:07 +0800
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"my-csi-app","namespace":"default"},"spec":{"containers":[{"command":[...
Status:       Pending
IP:           
IPs:          <none>
Containers:
  my-frontend:
    Container ID:  
    Image:         busybox
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      sleep
      1000000
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /data from seaweedfs-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-hhsgv (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  seaweedfs-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  seaweedfs-csi-pvc
    ReadOnly:   false
  default-token-hhsgv:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-hhsgv
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason              Age                 From                     Message
  ----     ------              ----                ----                     -------
  Normal   Scheduled           12m                 default-scheduler        Successfully assigned default/my-csi-app to k8s-master-27
  Warning  FailedMount         4m4s (x4 over 10m)  kubelet, k8s-master-27   Unable to attach or mount volumes: unmounted volumes=[seaweedfs-volume], unattached volumes=[seaweedfs-volume default-token-hhsgv]: timed out waiting for the condition
  Warning  FailedMount         108s                kubelet, k8s-master-27   Unable to attach or mount volumes: unmounted volumes=[seaweedfs-volume], unattached volumes=[default-token-hhsgv seaweedfs-volume]: timed out waiting for the condition
  Warning  FailedAttachVolume  38s (x14 over 12m)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131" : csinode.storage.k8s.io "k8s-master-27" not found
root@k8s-master-27:~/seaweedfs-csi-driver-1.0.9# kubectl get csinode.storage.k8s.io 
NAME            CREATED AT
k8s-master-27   2022-05-10T07:40:22Z
k8s-slave-28    2022-05-10T07:40:40Z
root@k8s-master-27:~/seaweedfs-csi-driver-1.0.9#
root@k8s-master-27:~/seaweedfs-csi-driver-1.0.9# kubectl get pvc
NAME                STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        AGE
seaweedfs-csi-pvc   Bound    pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131   5Gi        RWO            seaweedfs-storage   14m
root@k8s-master-27:~/seaweedfs-csi-driver-1.0.9# 
kubectl -n kube-system logs -f csi-seaweedfs-controller-0 -c csi-provisioner 
I0510 08:08:46.032012       1 controller.go:753] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:seaweedfs-csi-driver,VolumeHandle:pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131,ReadOnly:false,FSType:,VolumeAttributes:map[string]string{storage.kubernetes.io/csiProvisionerIdentity: 1652170105140-8081-seaweedfs-csi-driver,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,ControllerExpandSecretRef:nil,}}
I0510 08:08:46.032453       1 controller.go:1420] provision "default/seaweedfs-csi-pvc" class "seaweedfs-storage": volume "pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131" provisioned
I0510 08:08:46.032602       1 controller.go:1437] provision "default/seaweedfs-csi-pvc" class "seaweedfs-storage": succeeded
I0510 08:08:46.032751       1 volume_store.go:154] Saving volume pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131
I0510 08:08:46.048391       1 volume_store.go:157] Volume pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131 saved
E0510 08:08:46.048666       1 controller.go:1443] couldn't create key for object pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131: object has no meta: object does not implement the Object interfaces
I0510 08:08:46.048844       1 controller.go:1078] Claim processing succeeded, removing PVC acd523c8-8c7e-4e0f-804f-57041ab9f131 from claims in progress
I0510 08:08:46.048949       1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"seaweedfs-csi-pvc", UID:"acd523c8-8c7e-4e0f-804f-57041ab9f131", APIVersion:"v1", ResourceVersion:"3235773", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-acd523c8-8c7e-4e0f-804f-57041ab9f131
b1r3k commented 10 months ago

Maybe it will help you: check if you have DaemonSet seaweedfs-node created