Quickstart setup - "permission denied" in PVC-using pod

[GBBx]

What steps did you take and what happened: I created a small cluster of just two machines (k0s), I went through the quickstart and created a MariaDB container that uses the PVC. It runs for a few days or weeks but then it starts crashing saying

/usr/sbin/mysqld: Can't change dir to '/var/lib/mysql/' (Errcode: 13 "Permission denied")

Interestingly I have:

• 2 nodes
• 1 PVC
• 2 PVs with default/example-jiva-csi-pvc claim
• 3 JVs

I think one of the PVs and one of the JVs got created again but I don't know why.

What did you expect to happen: I expect the pod to be able to use the mounted PVC.

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other pastebin is fine.)

kubectl logs -n openebs jiva-operator-7df8bb6f9c-slvf9
I0323 08:08:25.432240       1 request.go:655] Throttling request took 1.047684099s, request: GET:https://172.18.0.1:443/apis/admissionregistration.k8s.io/v1?timeout=32s
time="2022-03-23T08:08:25Z" level=info msg="Go Version: go1.14.7"
time="2022-03-23T08:08:25Z" level=info msg="Go OS/Arch: linux/amd64"
time="2022-03-23T08:08:25Z" level=info msg="Version of jiva-operator: 3.1.0"

apiVersion: openebs.io/v1
kind: JivaVolume
metadata:
  annotations:
    openebs.io/volume-policy: example-jivavolumepolicy
  creationTimestamp: "2022-02-01T11:36:51Z"
  generation: 64
  labels:
    nodeID: <secondary_node>
    openebs.io/component: jiva-volume
    openebs.io/persistent-volume: pvc-219fc6d9-d0d4-4524-ab82-6e15be243c21
    openebs.io/persistent-volume-claim: example-jiva-csi-pvc
  name: pvc-219fc6d9-d0d4-4524-ab82-6e15be243c21
  namespace: openebs
  resourceVersion: "6753689"
  uid: d4dca3af-df2e-476b-b330-e6e39db6bbbd
spec:
  accessType: mount
  capacity: 1Gi
  desiredReplicationFactor: 1
  iscsiSpec:
    iqn: iqn.2016-09.com.openebs.jiva:pvc-219fc6d9-d0d4-4524-ab82-6e15be243c21
    targetIP: 172.18.239.191
    targetPort: 3260
  mountInfo:
    devicePath: /dev/disk/by-path/ip-172.18.239.191:3260-iscsi-iqn.2016-09.com.openebs.jiva:pvc-219fc6d9-d0d4-4524-ab82-6e15be243c21-lun-0
    fsType: ext4
    stagingPath: /var/lib/k0s/kubelet/plugins/kubernetes.io/csi/pv/pvc-219fc6d9-d0d4-4524-ab82-6e15be243c21/globalmount
  policy:
    replica:
      resources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/notReady
        operator: Exists
      - effect: NoExecute
        key: node.cloudprovider.kubernetes.io/uninitialized
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/unschedulable
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/out-of-disk
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/memory-pressure
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/disk-pressure
        operator: Exists
      - effect: NoExecute
        key: node.kubernetes.io/network-unavailable
        operator: Exists
    replicaSC: openebs-hostpath
    target:
      auxResources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      replicationFactor: 1
      resources:
        limits:
          cpu: "0"
          memory: "0"
        requests:
          cpu: "0"
          memory: "0"
      tolerations:
      - effect: NoExecute
        key: node.kubernetes.io/notReady
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.kubernetes.io/unreachable
        operator: Exists
        tolerationSeconds: 0
      - effect: NoExecute
        key: node.kubernetes.io/not-ready
        operator: Exists
        tolerationSeconds: 0
  pv: pvc-219fc6d9-d0d4-4524-ab82-6e15be243c21
status:
  phase: Ready
  replicaCount: 1
  replicaStatus:
  - address: tcp://172.17.1.176:9502
    mode: RW
  status: RW
versionDetails:
  desired: 3.1.0
  status:
    current: 3.1.0
    dependentsUpgraded: true
    lastUpdateTime: null

kubectl get jvp -n openebs -o yaml
apiVersion: v1
items:
- apiVersion: openebs.io/v1
  kind: JivaVolumePolicy
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"openebs.io/v1alpha1","kind":"JivaVolumePolicy","metadata":{"annotations":{},"name":"example-jivavolumepolicy","namespace":"openebs"},"spec":{"replicaSC":"openebs-hostpath","target":{"replicationFactor":1}}}
    creationTimestamp: "2022-01-19T14:16:33Z"
    generation: 1
    name: example-jivavolumepolicy
    namespace: openebs
    resourceVersion: "1282818"
    uid: 0ae414c3-2a21-4ba8-b354-3643576947b4
  spec:
    replicaSC: openebs-hostpath
    target:
      replicationFactor: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Environment:

• Jiva version: 3.1.0
• OpenEBS version: 3.0.0
• Kubernetes version (use kubectl version):

  kubectl version
  Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"\"not_available\"", GitTreeState:"", BuildDate:"2021-11-30T17:30:31Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4+k0s", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-30T17:23:13Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:

  k0s version
  v1.22.4+k0s.1

• Cloud provider or hardware configuration: on premises
• OS (e.g. from /etc/os-release): AlmaLinux