Questions on multiple pods accessing same PVC/PV (ReadWriteOnce) using local path provisioner

I have created a PV with the following yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ $.Release.Name }}-db-primary-data-pv
  labels:
    component: primary-data
    app.kubernetes.io/component: primary-data
spec:
  storageClassName: local-path
  capacity:
    storage: {{ $.Values.primary.persistence.data.size }}
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  local:
    path: {{ tpl .Values.primary.persistence.data.hostPath . }}
  nodeAffinity: 
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: {{ $.Values.global.pv.nodeAffinity.selector.key }}
              operator: In
              values:
                -  {{ $.Values.global.pv.nodeAffinity.selector.value }}

and this PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ $.Release.Name }}-db-primary-data-pvc
  labels:
    component: primary-data
    app.kubernetes.io/component: primary-data
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: {{ $.Values.primary.persistence.data.size }}
  storageClassName: local-path
  selector:
    matchLabels:
      component: primary-data

And I want to create two pods, e.g. mysql and mysql-backup, both pods are running in the same node, to be able to access the PV, in the same node as the pods, for accessing mysql data-dir at the same time. both pods are using the same PVC. Will that work?

I have created two pods in k3s with volume using the same PVC. In both pods (deployed using helm twice, with different release name), I can see the directory. but only one of the pod can browse the directory whereas the other one encountered 'permission denied' error.

storage class


kind: StorageClass
metadata:
annotations:
defaultVolumeType: local
objectset.rio.cattle.io/applied: <>
objectset.rio.cattle.io/id: ""
objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
objectset.rio.cattle.io/owner-name: local-storage
objectset.rio.cattle.io/owner-namespace: kube-system
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2024-06-12T12:10:55Z"
labels:
objectset.rio.cattle.io/hash: <hash>
name: local-path
resourceVersion: "215"
uid: 2be93f17-e37f-4537-8af3-a17dabb411e2
provisioner: rancher.io/local-path
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

- PVC
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: fb-store
    meta.helm.sh/release-namespace: rmf
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volumeType: local
  creationTimestamp: "2024-07-18T21:51:30Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: fb-store-fb-db
    app.kubernetes.io/managed-by: Helm
  name: fb-store-fb-db-pvc
  namespace: rmf
  resourceVersion: "2790323"
  uid: 38eb3f70-db76-489d-b1a1-939c3a78920d
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Mi
  selector:
    matchLabels:
      app: fb-store-fb-db
  storageClassName: local-path
  volumeMode: Filesystem
  volumeName: fb-store-fb-db-pv
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Mi
  phase: Bound

apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
meta.helm.sh/release-name: fb-store
meta.helm.sh/release-namespace: rmf
pv.kubernetes.io/bound-by-controller: "yes"
creationTimestamp: "2024-07-18T21:51:30Z"
finalizers:
- kubernetes.io/pv-protection
labels:
app: fb-store-fb-db
app.kubernetes.io/managed-by: Helm
name: fb-store-fb-db-pv
resourceVersion: "2790320"
uid: 19cacc7c-1d66-4f24-89b5-12d359a198d5
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Mi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: fb-store-fb-db-pvc
namespace: rmf
resourceVersion: "2788306"
uid: 38eb3f70-db76-489d-b1a1-939c3a78920d
local:
path: /romih_t1/fb/db
nodeAffinity:
required:
  nodeSelectorTerms:
  - matchExpressions:
    - key: kubernetes.io/hostname
      operator: In
      values:
      - <hostname>
persistentVolumeReclaimPolicy: Retain
storageClassName: local-path
volumeMode: Filesystem
status:
lastPhaseTransitionTime: "2024-07-18T22:00:35Z"
phase: Bound

pod yaml

...
volumeMounts:
- mountPath: /db
  name: db
...
volumes:
- name: db
persistentVolumeClaim:
  claimName: fb-store-fb-db-pvc
...

folder permission in container in both pods:

drwxrws---    2 1000     2000            25 Jul 18 22:06 db
drwxrws---    2 1000     2000            21 Jul 18 22:08 rootdir

folder permission in host:

drwxrws---. 2 username 2000 25 Jul 19 06:06 db
drwxrws---. 2 username 2000 21 Jul 19 06:08 rootDir

i also tested using the following:

PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: test-pvc
annotations:
volumeType: local
spec:
accessModes:
- ReadWriteOnce
storageClassName: local-path
resources:
requests:
  storage: 10Mi

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: demo
name: busybox
spec:
replicas: 5
selector:
matchLabels:
  app: demo
template:
metadata:
  labels:
    app: demo
spec:
  volumes:
  - name: vol
    persistentVolumeClaim:
      claimName: test-pvc
  terminationGracePeriodSeconds: 5
  containers:
  - image: docker.io/rancher/mirrored-library-busybox:1.36.1
    imagePullPolicy: IfNotPresent
    name: busybox
    volumeMounts:
    - name: vol
      mountPath: "/mnt"
    command:
      - sh
    args:
      - -c
      - |
        while true; do
          echo "$(date) $(hostname)" > /mnt/shared.log
          sleep $(($RANDOM % 5 + 5))
        done
    securityContext:
      seLinuxOptions: {}
      runAsUser: 1000
      runAsGroup: 2000
      runAsNonRoot: true
      readOnlyRootFilesystem: false
      allowPrivilegeEscalation: false
      privileged: false
      capabilities:
        drop: ["ALL"]
      seccompProfile:
        type: "RuntimeDefault"

only one of the pod is able to write to /mnt/shared.log

deployed pvc yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
  {"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{"volumeType":"local"},"name":"test-pvc","namespace":"rmf"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Mi"}},"storageClassName":"local-path"}}
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volumeType: local
creationTimestamp: "2024-07-19T04:02:16Z"
finalizers:
- kubernetes.io/pvc-protection
name: test-pvc
namespace: rmf
resourceVersion: "2861033"
uid: acb2fb1f-4a49-45d2-92a7-21647cfb297a
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
  storage: 10Mi
storageClassName: local-path
volumeMode: Filesystem
volumeName: pvc-51290e8e-9b18-45bb-b0e7-2a726208eed8
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Mi
phase: Bound

provisioned pv yaml:


apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
local.path.provisioner/selected-node: <hostname>
pv.kubernetes.io/provisioned-by: rancher.io/local-path
creationTimestamp: "2024-07-19T03:59:35Z"
finalizers:
- kubernetes.io/pv-protection
name: pvc-51290e8e-9b18-45bb-b0e7-2a726208eed8
resourceVersion: "2861031"
uid: 99adc40e-fff7-4a91-b20e-56710324817d
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Mi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: test-pvc
namespace: rmf
resourceVersion: "2861029"
uid: acb2fb1f-4a49-45d2-92a7-21647cfb297a
local:
path: /var/lib/rancher/k3s/storage/pvc-51290e8e-9b18-45bb-b0e7-2a726208eed8_rmf_test-pvc
nodeAffinity:
required:
  nodeSelectorTerms:
  - matchExpressions:
    - key: kubernetes.io/hostname
      operator: In
      values:
      - <hostname>
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
volumeMode: Filesystem
status:
lastPhaseTransitionTime: "2024-07-19T04:02:16Z"
phase: Bound


I also notice this event:

running PreBind plugin "VolumeBinding": Operation cannot be fulfilled on persistentvolumeclaims "test-pvc": the object has been modified; please apply your changes to the latest version and try again

rancher / local-path-provisioner

Questions on multiple pods accessing same PVC/PV (ReadWriteOnce) using local path provisioner #434