mkdir: cannot create directory '/bitnami/mariadb': Permission denied

[kjellmoens]

Which chart: bitnami/mariab

Describe the bug

mariadb 12:47:08.65
mariadb 12:47:08.65 Welcome to the Bitnami mariadb container
mariadb 12:47:08.65 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
mariadb 12:47:08.65 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
mariadb 12:47:08.65
mariadb 12:47:08.65 INFO  ==> ** Starting MariaDB setup **
mariadb 12:47:08.68 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
mariadb 12:47:08.69 INFO  ==> Initializing mariadb database
mkdir: cannot create directory '/bitnami/mariadb': Permission denied

To Reproduce

helm install --set primary.service.type=LoadBalancer --set primary.persistence.existingClaim=pvc-mariadb-data  --set volumePermissions.enabled=true mariadb bitnami/mariadb -n mariadb

Expected behavior The datastore of the database is correctly created

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.7.0", GitCommit:"eeac83883cb4014fe60267ec6373570374ce770b", GitTreeState:"clean", GoVersion:"go1.16.8"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Additional context

Name:         mariadb-0
Namespace:    mariadb
Priority:     0
Node:         node1/192.168.2.16
Start Time:   Tue, 04 Jan 2022 13:10:32 +0100
Labels:       app.kubernetes.io/component=primary
              app.kubernetes.io/instance=mariadb
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=mariadb
              controller-revision-hash=mariadb-67bb89b7b
              helm.sh/chart=mariadb-10.2.0
              statefulset.kubernetes.io/pod-name=mariadb-0
Annotations:  checksum/configuration: 0a2578eddd7792304887927962049d9d54511059a9a71626a98bd239aa2007e8
              cni.projectcalico.org/containerID: d21405dcfa3377d14fe43b86b5e1ba6a0e3fb5cd1bce299a11f11c7d25364f0e
              cni.projectcalico.org/podIP: 10.233.90.5/32
              cni.projectcalico.org/podIPs: 10.233.90.5/32
Status:       Running
IP:           10.233.90.5
IPs:
  IP:           10.233.90.5
Controlled By:  StatefulSet/mariadb
Init Containers:
  volume-permissions:
    Container ID:  containerd://29b9c5faf4ccaadc79e645005dfedda4a2789c2e8d4a84f951e1551e3d33eb2b
    Image:         docker.io/bitnami/bitnami-shell:10-debian-10-r279
    Image ID:      docker.io/bitnami/bitnami-shell@sha256:b9dba2d6bf011513a8914bdbd2c3307c9253d235d2c50d158b5d1c1359a6fe39
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/bash
      -ec
      chown -R 1001:1001 /bitnami/mariadb

    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 04 Jan 2022 13:10:33 +0100
      Finished:     Tue, 04 Jan 2022 13:10:33 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /bitnami/mariadb from data (rw)
Containers:
  mariadb:
    Container ID:   containerd://ba51d12c949ecda31ba0661810444bb3e78c04cbdef543bcd60504db332b184d
    Image:          docker.io/bitnami/mariadb:10.5.13-debian-10-r32
    Image ID:       docker.io/bitnami/mariadb@sha256:4969eda3a6cbb8007b4e52992979d9e1f1685cbed7c21afd2b44a64797c9e400
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 04 Jan 2022 13:47:08 +0100
      Finished:     Tue, 04 Jan 2022 13:47:08 +0100
    Ready:          False
    Restart Count:  12
    Liveness:       exec [/bin/bash -ec password_aux="${MARIADB_ROOT_PASSWORD:-}"
if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=120s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/bash -ec password_aux="${MARIADB_ROOT_PASSWORD:-}"
if [[ -f "${MARIADB_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MARIADB_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=30s timeout=1s period=10s #success=1 #failure=3
    Environment:
      BITNAMI_DEBUG:          false
      MARIADB_ROOT_PASSWORD:  <set to the key 'mariadb-root-password' in secret 'mariadb'>  Optional: false
      MARIADB_DATABASE:       my_database
    Mounts:
      /bitnami/mariadb from data (rw)
      /opt/bitnami/mariadb/conf/my.cnf from config (rw,path="my.cnf")
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      mariadb
    Optional:  false
  data:
    Type:        PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:   pvc-mariadb-data
    ReadOnly:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason            Age                 From            Message
  ----     ------            ----                ----            -------
  Warning  FailedScheduling  <unknown>                           0/2 nodes are available: 2 persistentvolumeclaim "pvc-mariadb-data" not found.
  Warning  FailedScheduling  <unknown>                           0/2 nodes are available: 2 persistentvolumeclaim "pvc-mariadb-data" not found.
  Normal   Scheduled         <unknown>                           Successfully assigned mariadb/mariadb-0 to node1
  Normal   Pulled            39m                 kubelet, node1  Container image "docker.io/bitnami/bitnami-shell:10-debian-10-r279" already present on machine
  Normal   Created           39m                 kubelet, node1  Created container volume-permissions
  Normal   Started           39m                 kubelet, node1  Started container volume-permissions
  Normal   Pulled            38m (x4 over 39m)   kubelet, node1  Container image "docker.io/bitnami/mariadb:10.5.13-debian-10-r32" already present on machine
  Normal   Created           38m (x4 over 39m)   kubelet, node1  Created container mariadb
  Normal   Started           38m (x4 over 39m)   kubelet, node1  Started container mariadb
  Warning  BackOff           4m (x180 over 39m)  kubelet, node1  Back-off restarting failed container

[carrodher]

Note the MariaDB container is a non-root container , because of that the directory (or volume) where the container needs to write data or create dirs should have the proper permissions. In this case, it seems the directory is trying to mount doesn't have the proper permission to work with non-root containers.

You can modify the permission of the volume or change the security context for the container/pod to run the container as a privileged user, it will depend on the policy you can/would like to apply.

[kjellmoens]

Hi,

I included the setting --set volumePermissions.enabled=true and you see a container volume-permissions being run

Normal   Created           39m                 kubelet, node1  Created container volume-permissions
Normal   Started           39m                 kubelet, node1  Started container volume-permissions

And when I look at the permissions they are correct

total 0
drwxrwxrwx+ 1 1001 1001 366 Jan  4 13:10 .
drwxrwxrwx+ 1 1001 1001  64 Jan  2 13:08 ..
drwxrwxrwx+ 1 1001 1001   0 Jan  4 13:10 mariadb-pvc-mariadb-data-pvc-6735ec69-de3b-43d3-9132-ebe93e5cd8d7

But still is receive permission denied

[carrodher]

It's really weird, I'm not able to reproduce the issue on my side but maybe I'm creating the PVC in a different way. Can you please provide us with the steps/manifests you're using to create the PVC that is being used as an existing claim?

[SQLJames]

I ran into this issue just deploying the default chart, no prior steps. The only values I deployed with are below:

• wordpressUsername:
• wordpressPassword:
• wordpressEmail:
• wordpressFirstName:
• wordpressLastName:
• wordpressBlogName:

[carrodher]

@SQLJames please, can you open a new issue? It seems you're using bitnami/wordpress while this issue is about bitnami/mariadb

[gsaslis]

+1 hitting this issue with the mariadb chart also. :/

[gsaslis]

@carrodher in my case, the PVC is being created with the https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner, by setting primary.persistence.storageClass=managed-nfs-storage when installing the chart.

[kjellmoens]

Same for me. I using also nfs-subdir-external-provisioner.

On Sun, Jan 23, 2022, 07:12 Yorgos Saslis @.***> wrote:

@carrodher https://github.com/carrodher in my case, the PVC is being created with the https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner, by setting primary.persistence.storageClass=managed-nfs-storage when installing the chart.

— Reply to this email directly, view it on GitHub https://github.com/bitnami/charts/issues/8565#issuecomment-1019423091, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACAZ4EEJILIQZ33QFVPMWW3UXOL2LANCNFSM5LHL6IBA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you authored the thread.Message ID: @.***>

[carrodher]

Can you check with ls -la what are the permissions of the volume/folder inside and outside the container? In the same way, can you also set the image.debug=true parameter so there is more information in the container log about the initialization process?

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[crackers8199]

this can be resolved with the mariadb chart by using the following initContainer:

primary:
  initContainers:
    - name: mariadb-create-directory-structure
      image: busybox
      command:
        [
          "sh",
          "-c",
          "/bin/mkdir -p /bitnami/mariadb/data && /bin/chmod -R 777 /bitnami/mariadb",
        ]
      volumeMounts:
        - name: data
          mountPath: /bitnami/mariadb/data

not the most secure way of doing things obviously, but it's working for me for local development at least. probably a better idea is to try to change the user rather than just opening the whole directory up to everyone.

also, unfortunately that doesn't help with the wordpress chart, as there's no way to fix it that i've seen there. if anyone has any ideas, i'm all ears.

[akbarkn]

I faced this issue in Kubernetes v1.26.x and figured out it happened because the volume ownership modification (fsGroup: 1001) didn't work.

Based on this docs, I changed that value of fsGroupPolicy to File in CSIDriver and it worked.

[bronius-congruity]

I'll add two cents: For me, starting up a recipe referencing mariadb begets this error on an existing lando-run project after upgrading to latest lando 3.21 beta. Using the same sort of recipe, I can start a new project (and db comes up, and healthcheck passes), but I can't seem to latch onto existing ones. lando logs -s database

Lando Log

``` database_1 | userperms 21:23:33.29 WARN ==> Only the root user can reset permissions! This is probably ok though... database_1 | loadkeys 21:23:33.39 WARN ==> Only the root user can load ssh keys! This is probably ok though... database_1 | lando 21:23:33.40 INFO ==> Lando handing off to: /launch.sh database_1 | lando 21:23:33.42 DEBUG ==> Running command with exec... database_1 | mariadb 21:23:33.51 database_1 | mariadb 21:23:33.54 Welcome to the Bitnami mariadb container database_1 | mariadb 21:23:33.55 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb database_1 | mariadb 21:23:33.57 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues database_1 | mariadb 21:23:33.59 database_1 | mariadb 21:23:33.71 INFO ==> ** Starting MariaDB setup ** database_1 | mariadb 21:23:33.80 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars database_1 | mariadb 21:23:33.83 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment. database_1 | mariadb 21:23:33.88 INFO ==> Initializing mariadb database database_1 | mariadb 21:23:33.91 DEBUG ==> Ensuring expected directories/files exist database_1 | mkdir: cannot create directory '/bitnami/mariadb/data': Permission denied ```

Previous version (3.20) supported and knew what to do with pantheon-mariadb as my database but not beta. So I used mariadb:10.4 in the new version. Unsure where/how to apply permissions as some of the comments above propose, but willing to try if it will help me and anyone more generally.

Two more cents:

• Reverting to lando 3.20 (latest stable and previous version and version that supports database: pantheon-mariadb), the same mariadb:10.4 still suffers the same permission snag
• But also reverting to database: pantheon-mariadb does come up without a hitch ¯_(ツ)_/¯