GRPC error: Error setting bucket metadata: mkdir /buckets/ CreateEntry: rpc error: code = Unavailable desc = connection closed

[aoberest]

Hello. I have a working SeaweedFS cluster with "Security Configuration" I have grpc_ca tls cert and grpc.client tls cert and key

I used 'helm chart' and start seaweedfs-csi-driver.

helm install --set seaweedfsFiler=10.193.86.149:8888 --set tlsSecret=seaweedfs-csi-tls seaweedfs-csi-driver ./deploy/helm/seaweedfs-csi-driver

security.toml

# Put this file to one of the location, with descending priority
#    ./security.toml
#    $HOME/.seaweedfs/security.toml
#    /etc/seaweedfs/security.toml
# this file is read by master, volume server, and filer

[grpc]
ca = "/var/run/secrets/app/tls/ca.crt"

[grpc.volume]
cert = "/usr/local/share/ca-certificates/volume/tls.crt"
key  = "/usr/local/share/ca-certificates/volume/tls.key"

[grpc.master]
cert = "/usr/local/share/ca-certificates/master/tls.crt"
key  = "/usr/local/share/ca-certificates/master/tls.key"

[grpc.filer]
cert = "/usr/local/share/ca-certificates/filer/tls.crt"
key  = "/usr/local/share/ca-certificates/filer/tls.key"

# use this for any place needs a grpc client
# i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
[grpc.client]
cert = "/var/run/secrets/app/tls/tls.crt"
key  = "/var/run/secrets/app/tls/tls.key"

kubectl logs seaweedfs-csi-driver-controller-0 seaweedfs-csi-plugin

I0629 06:05:58 1 main.go:36] connect to filer 10.193.86.149:8888
I0629 06:05:58 1 driver.go:47] Driver: seaweedfs-csi-driver version: 1.0.0
I0629 06:05:58 1 driver.go:93] Enabling volume access mode: SINGLE_NODE_WRITER
I0629 06:05:58 1 driver.go:93] Enabling volume access mode: MULTI_NODE_MULTI_WRITER
I0629 06:05:58 1 driver.go:104] Enabling controller service capability: CREATE_DELETE_VOLUME
I0629 06:05:58 1 driver.go:104] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0629 06:05:58 1 server.go:92] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I0629 06:05:58 1 filer_client.go:211] mkdir directory:"/buckets" entry:{name:"pvc-37c13912-818d-4918-9fb1-e1c2087978c4" is_directory:true attributes:{mtime:1624946758 file_mode:2147484159 crtime:1624946758}}: CreateEntry: rpc error: code = Unavailable desc = connection closed
E0629 06:05:58 1 utils.go:56] GRPC error: Error setting bucket metadata: mkdir /buckets/pvc-37c13912-818d-4918-9fb1-e1c2087978c4: CreateEntry: rpc error: code = Unavailable desc = connection closed
I0629 06:05:59 1 filer_client.go:211] mkdir directory:"/buckets" entry:{name:"pvc-37c13912-818d-4918-9fb1-e1c2087978c4" is_directory:true attributes:{mtime:1624946759 file_mode:2147484159 crtime:1624946759}}: CreateEntry: rpc error: code = Unavailable desc = connection closed
E0629 06:05:59 1 utils.go:56] GRPC error: Error setting bucket metadata: mkdir /buckets/pvc-37c13912-818d-4918-9fb1-e1c2087978c4: CreateEntry: rpc error: code = Unavailable desc = connection closed

How to fix GRPC error ?

Could you please help me ? @chrislusf @ustuzhanin

[aoberest]

./seaweedfs-csi-driver -version { "driverVersion": "", "gitCommit": "bb90a8b", "buildDate": "", "goVersion": "go1.16", "compiler": "gc", "platform": "linux/amd64" }

[ustuzhanin]

Check listen ports on filer. In my setup seaweedfsFiler: "seaweedfs-fuse-api.kube-throne.svc:9090"

[aoberest]

Thank for your answer. @ustuzhanin Do you have a working seaweedfs cluster inside k8s - ?

In my case, i have a standalone seaweedfs cluster. on my filer:

filer]# ss -ntpl | grep 8888
LISTEN     0      128         :::18888                   :::*                   users:(("weed",pid=77228,fd=16))
LISTEN     0      128         :::8888                    :::*                   users:(("weed",pid=77228,fd=15))

I think that grpc security configuration does't work on client. I add in statefulset.yml and I guess seaweedfs-csi-plugin should use it

volumeMounts:
  - name: security-config
    readOnly: true
    mountPath: /etc/seaweedfs/security.toml
    subPath: security.toml

https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/driver.go#L49 must be load /etc/seaweedfs/security.toml config, right ? @chrislusf

[aoberest]

How to properly connect with grpc tls client certificates to the cluster ? Do you have instructions? @chrislusf

[ustuzhanin]

https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/deploy/helm/seaweedfs-csi-driver/templates/statefulset.yml#L87 In this mount TLS into container https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/deploy/helm/seaweedfs-csi-driver/templates/statefulset.yml#L69 In this add ENV to use TLS

All values:

# host and port of your SeaweedFs filer
seaweedfsFiler: "seaweedfs-fuse-api.kube-throne.svc:9090"
isDefaultStorageClass: false
tlsSecret: "seaweedfs-fuse-cert"

imagePullPolicy: "IfNotPresent"

csiProvisioner:
  image: <our private registry>/library/k8scsi/csi-provisioner:v2.0.4
  resources:
    limits: {memory: "128Mi", cpu: "0.5"}
    requests: {memory: "32Mi", cpu: "0.1"}

csiAttacher:
  image: <our private registry>/library/k8scsi/csi-attacher:v3.0.2
  resources:
    limits: {memory: "128Mi", cpu: "0.5"}
    requests: {memory: "32Mi", cpu: "0.1"}

csiNodeDriverRegistrar:
  image: <our private registry>/library/k8scsi/csi-node-driver-registrar:v2.0.1
  resources:
    limits: {memory: "128Mi", cpu: "0.5"}
    requests: {memory: "32Mi", cpu: "0.1"}

seaweedfsCsiPlugin:
  image: <our private registry>/library/chrislusf/seaweedfs-csi-driver:latest-2021-02-17
  resources:
    limits: {memory: "1Gi", cpu: "2"}
    requests: {memory: "512Mi", cpu: "0.5"}

node:
  # Deploy node daemonset
  enabled: true
  # When pod on node be recreated all pod on same node lost PV.
  # For safe update use updateStrategy.type: OnDelete and manual move pods who use PV and delete damonset pod
  updateStrategy:
    type: OnDelete

Do you have a working seaweedfs cluster inside k8s - ?

Yes 5 setups 3 in production. But we use only Object Storage over S3 CSI driver Did not fit. Applications for which they were going to use require posix locks on files

[ustuzhanin]

But we do not use https://github.com/chrislusf/seaweedfs/tree/master/k8s because we have a not quite standard k8s implementation. We keep a lot of independent clusters, and we use the consul as the SD and other things for network connectivity

And so everything is standard. filer + s3Api - Deployment, masters and volume servers with local disks - Statefulset

[ustuzhanin]

In seaweed cluster:

/ # cat /etc/seaweedfs/security.toml
# this file is read by master, volume server, and filer

# the jwt signing key is read by master and volume server
# a jwt expires in 10 seconds
[jwt.signing]
# key = ""
expires_after_seconds = 10           # seconds

[jwt.signing.read]
key = ""
expires_after_seconds = 10           # seconds
# all grpc tls authentications are mutual
# the values for the following ca, cert, and key are paths to the PERM files.
[grpc]
ca = "/usr/local/share/ca-certificates/ca.crt"

[grpc.volume]
cert = "/usr/local/share/ca-certificates/tls.crt"
key  = "/usr/local/share/ca-certificates/tls.key"
ca = "/usr/local/share/ca-certificates/ca.crt"

[grpc.master]
cert = "/usr/local/share/ca-certificates/tls.crt"
key  = "/usr/local/share/ca-certificates/tls.key"
ca = "/usr/local/share/ca-certificates/ca.crt"

[grpc.filer]
cert = "/usr/local/share/ca-certificates/tls.crt"
key  = "/usr/local/share/ca-certificates/tls.key"
ca = "/usr/local/share/ca-certificates/ca.crt"

[grpc.msg_broker]
cert = "/usr/local/share/ca-certificates/tls.crt"
key  = "/usr/local/share/ca-certificates/tls.key"
ca = "/usr/local/share/ca-certificates/ca.crt"

# use this for any place needs a grpc client
# i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
[grpc.client]
cert = "/usr/local/share/ca-certificates/tls.crt"
key  = "/usr/local/share/ca-certificates/tls.key"
ca = "/usr/local/share/ca-certificates/ca.crt"

# volume server https options
# Note: work in progress!
#     this does not work with other clients, e.g., "weed filer|mount" etc, yet.
[https.client]
enabled = false
[https.volume]
cert = ""
key  = ""/ #

[aoberest]

Thank you for sharing your experience.

But, if I understood correctly, I don't need security.toml for the seaweedfs-csi-driver to work, since the paths to tls client certificates are passed through environment variables. Are you sure this works for you?

[WhiteSash]

As far as I understand, a colleague has a version with a helm chart. And the option is not a helm, but a manual deployment from https://github.com/seaweedfs/seaweedfs-csi-driver/tree/master/deploy/kubernetes someone tested with enabled security?

[aoberest]

Yes. I tested it. But it doesn't work for me. Try it maybe it will work for you?

Prerequisites: Already have a working Kubernetes cluster (includes kubectl) Already have a working SeaweedFS cluster with enabled security. You must have a client certificate

Add the secret to k8s. kubectl -n kube-system create secret generic seaweedfs-csi-tls --from-file=ca.crt=cert/SeaweedFS_CA.crt --from-file=tls.crt=cert/client01.key --from-file=tls.key=cert/client01.crt

Create the file seaweedfs-csi-grpc.yaml

---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  creationTimestamp: null
  name: csinodeinfos.csi.storage.k8s.io
spec:
  group: csi.storage.k8s.io
  names:
    kind: CSINodeInfo
    plural: csinodeinfos
  scope: Cluster
  validation:
    openAPIV3Schema:
      properties:
        csiDrivers:
          description: List of CSI drivers running on the node and their properties.
          items:
            properties:
              driver:
                description: The CSI driver that this object refers to.
                type: string
              nodeID:
                description: The node from the driver point of view.
                type: string
              topologyKeys:
                description: List of keys supported by the driver.
                items:
                  type: string
                type: array
          type: array
  version: v1alpha1
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: seaweedfs-storage
  namespace: kube-system
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: seaweedfs-csi-driver
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-seaweedfs-controller-sa
  namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: csi-seaweedfs-node-sa
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-provisioner-role
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["get", "list"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-provisioner-binding
subjects:
  - kind: ServiceAccount
    name: csi-seaweedfs-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-seaweedfs-provisioner-role
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-attacher-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["volumeattachments/status"]
    verbs: ["patch"]

---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-attacher-binding
subjects:
  - kind: ServiceAccount
    name: csi-seaweedfs-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-seaweedfs-attacher-role
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-snapshotter-role
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create", "list", "watch", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-snapshotter-binding
subjects:
  - kind: ServiceAccount
    name: csi-seaweedfs-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-seaweedfs-snapshotter-role
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-driver-registrar-controller-role
rules:
  - apiGroups: ["csi.storage.k8s.io"]
    resources: ["csidrivers"]
    verbs: ["create", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-driver-registrar-controller-binding
subjects:
  - kind: ServiceAccount
    name: csi-seaweedfs-controller-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-seaweedfs-driver-registrar-controller-role
  apiGroup: rbac.authorization.k8s.io
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-driver-registrar-node-role
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create", "list", "watch", "delete"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: csi-seaweedfs-driver-registrar-node-binding
subjects:
  - kind: ServiceAccount
    name: csi-seaweedfs-node-sa
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: csi-seaweedfs-driver-registrar-node-role
  apiGroup: rbac.authorization.k8s.io
---
kind: StatefulSet
apiVersion: apps/v1
metadata:
  name: csi-seaweedfs-controller
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-seaweedfs-controller
  serviceName: "csi-seaweedfs"
  replicas: 1
  template:
    metadata:
      labels:
        app: csi-seaweedfs-controller
        role: csi-seaweedfs
    spec:
      priorityClassName: system-cluster-critical
      serviceAccount: csi-seaweedfs-controller-sa
      containers:
        # provisioner
        - name: csi-provisioner
          image: sig-storage/csi-provisioner:v2.0.4
          args:
            - "--csi-address=$(ADDRESS)"
            - "--v=5"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        # attacher
        - name: csi-attacher
          image: sig-storage/csi-attacher:v3.0.2
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--timeout=120s"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        # Driver registrar
        - name: csi-driver-registrar
          image: k8scsi/csi-cluster-driver-registrar:v1.0.1
          args:
            - "--v=5"
            - "--pod-info-mount-version=\"v1\""
            - "--csi-address=$(ADDRESS)"
          env:
            - name: ADDRESS
              value: /var/lib/csi/sockets/pluginproxy/csi.sock
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
        # SeaweedFs Plugin
        - name: seaweedfs-csi-plugin
          image: chrislusf/seaweedfs-csi-driver:latest
          args :
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--filer=$(SEAWEEDFS_FILER)"
            - "--nodeid=$(NODE_ID)"
            - "--cacheCapacityMB=$(SEAWEEDFS_CACHE_CAPACITY_MB)"
            - "--cacheDir=$(SEAWEEDFS_CACHE_DIR)"
          env:
            - name: CSI_ENDPOINT
              value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
            - name: SEAWEEDFS_FILER
              value: "10.193.86.149:8888" # host and port of your SeaweedFs filer
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: SEAWEEDFS_CACHE_CAPACITY_MB
              value: "1000"
            - name: SEAWEEDFS_CACHE_DIR
              value: /tmp
            - name: WEED_GRPC_CLIENT_KEY
              value: /var/run/secrets/app/tls/tls.key
            - name: WEED_GRPC_CLIENT_CERT
              value: /var/run/secrets/app/tls/tls.crt
            - name: WEED_GRPC_CA
              value: /var/run/secrets/app/tls/ca.crt
          imagePullPolicy: "Always"
          volumeMounts:
            - name: socket-dir
              mountPath: /var/lib/csi/sockets/pluginproxy/
            - name: tls
              mountPath: /var/run/secrets/app/tls
            - name: security-config
              mountPath: /etc/seaweedfs/security.toml
              subPath: security.toml
      volumes:
        - name: socket-dir
          emptyDir: {}
        - name: tls
          secret:
            secretName: seaweedfs-csi-tls
        - name: security-config
          configMap: 
            name: seaweedfs-csi-driver
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: csi-seaweedfs-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: csi-seaweedfs-node
  template:
    metadata:
      labels:
        app: csi-seaweedfs-node
        role: csi-seaweedfs
    spec:
      priorityClassName: system-node-critical
      serviceAccount: csi-seaweedfs-node-sa
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: driver-registrar
          image: sig-storage/csi-node-driver-registrar:v2.0.1
          args:
            - "--v=5"
            - "--csi-address=$(ADDRESS)"
            - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)"
          env:
            - name: ADDRESS
              value: /csi/csi.sock
            - name: DRIVER_REG_SOCK_PATH
              value: /var/lib/kubelet/plugins/seaweedfs-csi-driver/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi/
            - name: registration-dir
              mountPath: /registration/
        - name: csi-seaweedfs-plugin
          securityContext:
            privileged: true
            capabilities:
              add: ["SYS_ADMIN"]
            allowPrivilegeEscalation: true
          image: chrislusf/seaweedfs-csi-driver:latest
          args :
            - "--endpoint=$(CSI_ENDPOINT)"
            - "--filer=$(SEAWEEDFS_FILER)"
            - "--nodeid=$(NODE_ID)"
            - "--cacheCapacityMB=$(SEAWEEDFS_CACHE_CAPACITY_MB)"
            - "--cacheDir=$(SEAWEEDFS_CACHE_DIR)"
          env:
            - name: CSI_ENDPOINT
              value: unix:///csi/csi.sock
            - name: SEAWEEDFS_FILER
              value: "10.193.86.149:8888" # host and port of your SeaweedFs filer, could also be K8S DNS entry like seaweedfs-filer.seaweed-namespace.svc:8888
            - name: NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: SEAWEEDFS_CACHE_CAPACITY_MB
              value: "1000"
            - name: SEAWEEDFS_CACHE_DIR
              value: /tmp
            - name: WEED_GRPC_CLIENT_KEY
              value: /var/run/secrets/app/tls/tls.key
            - name: WEED_GRPC_CLIENT_CERT
              value: /var/run/secrets/app/tls/tls.crt
            - name: WEED_GRPC_CA
              value: /var/run/secrets/app/tls/ca.crt
          imagePullPolicy: "Always"
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: pods-mount-dir
              mountPath: /var/lib/kubelet
              mountPropagation: "Bidirectional"
            - mountPath: /dev
              name: device-dir
            - name: tls
              mountPath: /var/run/secrets/app/tls
            - name: security-config
              mountPath: /etc/seaweedfs/security.toml
              subPath: security.toml
      volumes:
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: DirectoryOrCreate
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/seaweedfs-csi-driver
            type: DirectoryOrCreate
        - name: pods-mount-dir
          hostPath:
            path: /var/lib/kubelet
            type: Directory
        - name: device-dir
          hostPath:
            path: /dev
        - name: tls
          secret:
            secretName: seaweedfs-csi-tls
        - name: security-config
          configMap: 
            name: seaweedfs-csi-driver
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: seaweedfs-csi-driver
  namespace: kube-system
  labels:
    app: seaweedfs-csi-driver
data:
  security.toml: |-
    # Put this file to one of the location, with descending priority
    #    ./security.toml
    #    $HOME/.seaweedfs/security.toml
    #    /etc/seaweedfs/security.toml
    # this file is read by master, volume server, and filer

    [grpc]
    ca = "/var/run/secrets/app/tls/ca.crt"

    # use this for any place needs a grpc client
    # i.e., "weed backup|benchmark|filer.copy|filer.replicate|mount|s3|upload"
    [grpc.client]
    cert = "/var/run/secrets/app/tls/tls.crt"
    key  = "/var/run/secrets/app/tls/tls.key"

and apply to k8s

kubectl apply -f deploy/kubernetes/seaweedfs-csi-grpc.yaml

Testing: Create a persistant volume claim for 5GiB with name seaweedfs-csi-pvc with storage class seaweedfs-storage. The value, 5Gib does not have any significance as for SeaweedFS the whole filesystem is mounted into the container. kubectl apply -f deploy/kubernetes/sample-seaweedfs-pvc.yaml Verify if the persistant volume claim exists and wait until its the STATUS is Bound kubectl get pvc

After that, i see error

kubectl -n kube-system logs csi-seaweedfs-controller-0 seaweedfs-csi-plugin

I0630 06:52:36 1 filer_client.go:231] mkdir directory:"/buckets" entry:{name:"pvc-d092019e-3e57-4617-a621-929017569013" is_directory:true attributes:{mtime:1625035956 file_mode:2147484159 crtime:1625035956}}: CreateEntry: rpc error: code = Unavailable desc = connection closed
E0630 06:52:36 1 utils.go:56] GRPC error: Error setting bucket metadata: mkdir /buckets/pvc-d092019e-3e57-4617-a621-929017569013: CreateEntry: rpc error: code = Unavailable desc = connection closed

Maybe I made a mistake in the settings or in the tls certificates and I don't see it. Try to repeat these steps at yourself. @WhiteSash