search

juicedata / juicefs-csi-driver

JuiceFS CSI Driver
https://github.com/juicedata/juicefs
Apache License 2.0
213 stars 81 forks source link

pathPattern not active #912

Open zhj-hx opened 5 months ago

zhj-hx commented 5 months ago

在storageclass配置了pathPattern后，CSI 驱动在 JuiceFS 创建的子目录名称还是 pvc-234bb954-dfa3-4251-9ebe-8727fb3ad6fd

Environment:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
provisioner: csi.juicefs.com
parameters:
  csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
  csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  juicefs/mount-cpu-limit: 2000m
  juicefs/mount-memory-limit: 5Gi
  juicefs/mount-cpu-request: 40m
  juicefs/mount-memory-request: 200Mi
  pathPattern: "${.PVC.namespace}-${.PVC.name}"
  #pathPattern: "${.pvc.namespace}-${.pvc.name}"
h0hmj commented 5 months ago

How did you deploy csi driver? feature pathPattern relies on juicefs provisioner, did you enable it?

https://juicefs.com/docs/zh/csi/guide/pv/#options-template

zhj-hx commented 5 months ago

我没注意到新版本的这个变化，已经把csi-provisioner，csi-resizer 这两个容器删了，现在juicefs-csi-controller配置如下，juicefs挂载的目录下还是pvc-234bb954-dfa3-4251-9ebe-8727fb3ad6fd

      containers:
      - args:
        - --endpoint=$(CSI_ENDPOINT)
        - --logtostderr
        - --nodeid=$(NODE_NAME)
        - --v=5
        - --provisioner=true
        env:
        - name: CSI_ENDPOINT
          value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: JUICEFS_MOUNT_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: JUICEFS_MOUNT_PATH
          value: /var/lib/juicefs/volume
        - name: JUICEFS_CONFIG_PATH
          value: /var/lib/juicefs/config
        image: juicedata/juicefs-csi-driver:v0.23.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: healthz
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        name: juicefs-plugin
        ports:
        - containerPort: 9909
          name: healthz
          protocol: TCP
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 150m
            memory: 512Mi
        securityContext:
          capabilities:
            add:
            - SYS_ADMIN
          privileged: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/csi/sockets/pluginproxy/
          name: socket-dir
        - mountPath: /jfs
          mountPropagation: Bidirectional
          name: jfs-dir
        - mountPath: /root/.juicefs
          mountPropagation: Bidirectional
          name: jfs-root-dir
      - args:
        - --csi-address=$(ADDRESS)
        - --health-port=$(HEALTH_PORT)
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: HEALTH_PORT
          value: "9909"
        image: quay.io/k8scsi/livenessprobe:v1.1.0
        imagePullPolicy: IfNotPresent
        name: liveness-probe
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /csi
          name: socket-dir
      dnsPolicy: ClusterFirst
      priorityClassName: system-cluster-critical
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: juicefs-csi-controller-sa
      serviceAccountName: juicefs-csi-controller-sa
      terminationGracePeriodSeconds: 30
      tolerations:
      - key: CriticalAddonsOnly
        operator: Exists
      volumes:
      - emptyDir: {}
        name: socket-dir
      - hostPath:
          path: /var/lib/juicefs/volume
          type: DirectoryOrCreate
        name: jfs-dir
      - hostPath:
          path: /var/lib/juicefs/config
          type: DirectoryOrCreate
        name: jfs-root-dir

我曾在kubernetes1.20版本部署juicefs-csi-driver v0.19版本，设置pathPattern 是正常的，今天我在现在这个集群测试juicefs-csi-driver v0.19也不生效...

h0hmj commented 5 months ago

ok. i will try to reproduce the problem with your version later

k8s v1.26.10 and csi driver 0.23.5

h0hmj commented 5 months ago

@zhj-hx hi, i tested your setup in my env. pathPattern works.

I recommand you to do a totally cleanup and install csi drive using helm with default values, recreate your sc and pvc.

Also try to adjust the loglevel of controller to 6, --v=6. During creating pvc using sc configured with pathPattern, you should see log Provisioner Resolved StorageClass.Parameters: xxx which contains generated pathPattern.

zhj-hx commented 5 months ago

@h0hmj conroller日志是这样的 I0402 02:58:08.776964 7 provisioner.go:117] Provisioner Resolved StorageClass.Parameters: map[csi.storage.k8s.io/node-publish-secret-name:juicefs-sc-secret csi.storage.k8s.io/node-publish-secret-namespace:kube-system csi.storage.k8s.io/provisioner-secret-name:juicefs-sc-secret csi.storage.k8s.io/provisioner-secret-namespace:kube-system]

zxh326 commented 5 months ago

这个改了 需要重建 pv 才能生效

有改过吗

zhj-hx commented 5 months ago

就是重建了之后的日志

zxh326 commented 5 months ago

就是重建了之后的日志

我的意思是需要删掉旧的 pv，创建新的pv pathPattern 才会生效。

如果你就是这么做的，可以贴一下 pv 的yaml 和 csi-node 的完整日志。

zhj-hx commented 5 months ago

是的，我将整个storageclass及juicefs-csi-driver都重建了 storageclass配置

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: juicefs-sc
provisioner: csi.juicefs.com
parameters:
  csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system
  csi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secret
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system
  juicefs/mount-cpu-limit: 2000m
  juicefs/mount-memory-limit: 5Gi
  juicefs/mount-cpu-request: 40m
  juicefs/mount-memory-request: 200Mi
  pathPattern: "${.PVC.namespace}-${.PVC.name}"
mountOptions:
  - cache-dir=/var/jfsCache
  - cache-size=5120
reclaimPolicy: Retain
volumeBindingMode: Immediate

pvc配置

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: juicefs-pvc
  namespace: default
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Pi
  storageClassName: juicefs-sc

csi-node日志

I0403 10:40:41.487736       7 node.go:107] NodePublishVolume: volume_id is pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7
I0403 10:40:41.487748       7 node.go:118] NodePublishVolume: volume_capability is mount:<fs_type:"juicefs" > access_mode:<mode:MULTI_NODE_MULTI_WRITER >
I0403 10:40:41.487778       7 node.go:124] NodePublishVolume: creating dir /var/lib/kubelet/pods/ec57cdfd-795d-423b-b86a-7a8b7388eebf/volumes/kubernetes.io~csi/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7/mount
I0403 10:40:41.487832       7 node.go:139] NodePublishVolume: volume context: map[capacity:11258999068426240 csi.storage.k8s.io/ephemeral:false csi.storage.k8s.io/node-publish-secret-name:juicefs-sc-secret csi.storage.k8s.io/node-publish-secret-namespace:kube-system csi.storage.k8s.io/pod.name:juicefs-app csi.storage.k8s.io/pod.namespace:default csi.storage.k8s.io/pod.uid:ec57cdfd-795d-423b-b86a-7a8b7388eebf csi.storage.k8s.io/provisioner-secret-name:juicefs-sc-secret csi.storage.k8s.io/provisioner-secret-namespace:kube-system csi.storage.k8s.io/serviceAccount.name:default subPath:pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7]
I0403 10:40:41.487848       7 node.go:149] NodePublishVolume: mounting juicefs with secret [storage access-key bucket metaurl name secret-key], options []
I0403 10:40:41.498548       7 juicefs.go:1033] ceFormat cmd: [/usr/local/bin/juicefs format --storage=tos --bucket=https://xxx.tos-cn-beijing.ivolces.com --access-key=xxx --secret-key=${secretkey} ${metaurl} juicefs]
I0403 10:40:41.515841       7 pod_mount.go:522] setMountLabel: set mount info in pod juicefs-app
I0403 10:40:41.526972       7 pod_mount.go:340] createOrAddRef: Need to create pod juicefs-172.16.0.3-pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop.
I0403 10:40:41.526986       7 common.go:125] ceMount: mount redis://:****@redis-xxx.redis.ivolces.com:6379/1 at /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop
I0403 10:40:41.526997       7 pod_mount.go:589] createOrUpdateSecret: juicefs-172.16.0.3-pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop-secret, kube-system
I0403 10:40:41.552554       7 pod_mount.go:403] mount path /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop not ready: stat /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop: no such file or directory
I0403 10:40:42.053200       7 pod_mount.go:403] mount path /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop not ready: stat /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop: no such file or directory
I0403 10:40:42.553371       7 pod_mount.go:403] mount path /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop not ready: stat /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop: no such file or directory
I0403 10:40:43.054025       7 pod_mount.go:403] mount path /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop not ready: stat /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop: no such file or directory
I0403 10:40:43.556582       7 pod_mount.go:409] Mount point /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop is ready
I0403 10:40:43.556601       7 juicefs.go:942] Mount: mounting "redis://:****@redis-xxx.redis.ivolces.com:6379/1" at "/jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop" with options []
I0403 10:40:43.570114       7 juicefs.go:192] CreateVol: volume not existed
I0403 10:40:43.574835       7 juicefs.go:245] BindTarget: binding /jfs/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7-bqcsop/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7 at /var/lib/kubelet/pods/ec57cdfd-795d-423b-b86a-7a8b7388eebf/volumes/kubernetes.io~csi/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7/mount
I0403 10:40:43.581069       7 juicefs.go:1033] ceFormat cmd: [/usr/local/bin/juicefs format --storage=tos --bucket=https://xxx.tos-cn-beijing.ivolces.com --access-key=xxx --secret-key=${secretkey} ${metaurl} juicefs]
I0403 10:40:43.581082       7 juicefs.go:847] SetQuota cmd: /usr/local/bin/juicefs quota set ${metaurl} --path pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7 --capacity 10485760
I0403 10:40:43.618139       7 juicefs.go:878] quota set success: 2024/04/03 10:40:43.608961 juicefs[277] <INFO>: Meta address: redis://:****@redis-xxx.redis.ivolces.com:6379/1 [interface.go:497]
2024/04/03 10:40:43.611992 juicefs[277] <WARNING>: maxmemory_policy is "volatile-lru",  we will try to reconfigure it to 'noeviction'. [redis.go:3583]
2024/04/03 10:40:43.612430 juicefs[277] <WARNING>: reconfigured maxmemory-policy to 'noeviction', but it's still volatile-lru [redis.go:3589]
2024/04/03 10:40:43.612599 juicefs[277] <INFO>: Ping redis latency: 153.628µs [redis.go:3600]
+------------------------------------------+--------+------+------+-----------+-------+-------+
|                   Path                   |  Size  | Used | Use% |   Inodes  | IUsed | IUse% |
+------------------------------------------+--------+------+------+-----------+-------+-------+
| pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7 | 10 PiB |  0 B |   0% | unlimited |     0 |       |
+------------------------------------------+--------+------+------+-----------+-------+-------+
I0403 10:40:43.618166       7 node.go:203] NodePublishVolume: mounted pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7 at /var/lib/kubelet/pods/ec57cdfd-795d-423b-b86a-7a8b7388eebf/volumes/kubernetes.io~csi/pvc-e6b0f439-9583-4c44-8326-0401d50ea4e7/mount with options []
zxh326 commented 5 months ago

是的，我将整个storageclass及juicefs-csi-driver都重建了

关键是 pv和 pvc 要删了重建才能生效....

zhj-hx commented 5 months ago

@zxh326 上面的日志是我新建的pvc的信息，你是说删除是指所有的pv及pvc? 我之前也是全部都删了重建的，并没有生效

vicaya commented 1 month ago

https://github.com/juicedata/juicefs-csi-driver/issues/931#issuecomment-2243993009