[bitnami/wordpress] 'Persisting WordPress installation' is stuck when use EFS persistent volume

burakovsky commented 2 years ago

Name and Version

bitnami/wordpress 13.1.3

What steps will reproduce the bug?

Create AWS EFS filesystem with the default configuration
I tested it only with an external database (AWS RDS)
install aws-efs-csi-driver

Create a storageClass for dynamic provisioning:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: efs-wordress
parameters:
provisioningMode: efs-ap
fileSystemId: FILESYSTEM_ID
directoryPerms: '2775'           ## also tried 700, 755, 777
uid: '1001'                      ## also tried default value: 50000 and root
gid: '1001'                      ## also tried default value: 50000 and 1001 in combination with `root` uid
provisioner: efs.csi.aws.com

Install Wordpress using Helm with custom values provided below.

Are you using any custom parameters or values?

custom values.yaml:

externalDatabase:
  host: EXTERNAL_DB_HOST
  port: 3306
  user: EXTERNAL_DB_USER
  password: EXTERNAL_DB_PASSWORD
  database: EXTERNAL_DB_NAME
persistence:
  enabled: true
  storageClass: efs-wordpress
  accessModes:
    - ReadWriteMany
volumePermissions:
  enabled: true
ingress:
  enabled: true
  ingressClassName: alb
  hostname: wordpress-domain.com
  path: /*
  annotations:
    alb.ingress.kubernetes.io/certificate-arn: AWS_CERT_ARN
    alb.ingress.kubernetes.io/success-codes: 200,301
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
wordpressUsername: WORDPRESS_USERNAME
wordpressPassword: WORDPRESS_PASSWORD
wordpressEmail: WORDPRESS_EMAIL

What is the expected behavior?

WordPress preparation steps will be finished successfully during the first run

What do you see instead?

Wordress installation is stuck on Persisting WordPress installation step until liveness probe restarts the container (readiness probes also failed) because Apache starts after WordPress preparation steps. After container restart, it became ready, because persisting step does not run after restart.

Additional information

I have 2 WordPress installations in my EKS cluster:

installation with one replica and EBS volume as a storage, which is created using default gp2 storage class
installation with one replica and EFS volume as a storage, which is created using aws-efs-csi-driver

Both installations use the same values. The difference is only in PersistentVolumeClaim configuration and different databases (hosted on the same host) are used.

The first installation works as expected, but when I used EFS storage, persisting Wordpress installation is not finish successfully. I don't see any errors in container logs even when container run with BITNAMI_DEBUG=true. I tried both

volumePermissions:
    enabled: true

and

volumePermissions:
    enabled: false

Also, I tried different mounting options (uid, gid, permissions), but the result was the same.

Final manifests (deployment and persistentVolumeClaim):

with EBS (works correctly):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-13.1.3
  name: wordpress-ebs
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: wordpress-ebs
      app.kubernetes.io/name: wordpress
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: wordpress-ebs
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: wordpress
        helm.sh/chart: wordpress-13.1.3
    spec:
      affinity: {}
      containers:
        - env:
            - name: BITNAMI_DEBUG
              value: "false"
            - name: ALLOW_EMPTY_PASSWORD
              value: "yes"
            - name: MARIADB_HOST
              value: <EXTERNAL_DB_HOST>
            - name: MARIADB_PORT_NUMBER
              value: "3306"
            - name: WORDPRESS_DATABASE_NAME
              value: <EXTERNAL_DB_NAME>
            - name: WORDPRESS_DATABASE_USER
              value: <EXTERNAL_DB_USER>
            - name: WORDPRESS_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: mariadb-password
                  name: wordpress-ebs-externaldb
            - name: WORDPRESS_USERNAME
              value: <WORDPRESS_USERNAME>
            - name: WORDPRESS_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: wordpress-password
                  name: wordpress-ebs
            - name: WORDPRESS_EMAIL
              value: <WORDPRESS_EMAIL>
            - name: WORDPRESS_FIRST_NAME
              value: FirstName
            - name: WORDPRESS_LAST_NAME
              value: LastName
            - name: WORDPRESS_HTACCESS_OVERRIDE_NONE
              value: "no"
            - name: WORDPRESS_ENABLE_HTACCESS_PERSISTENCE
              value: "no"
            - name: WORDPRESS_BLOG_NAME
              value: User's Blog!
            - name: WORDPRESS_SKIP_BOOTSTRAP
              value: "no"
            - name: WORDPRESS_TABLE_PREFIX
              value: wp_
            - name: WORDPRESS_SCHEME
              value: http
            - name: WORDPRESS_EXTRA_WP_CONFIG_CONTENT
            - name: WORDPRESS_AUTO_UPDATE_LEVEL
              value: none
            - name: WORDPRESS_PLUGINS
              value: none
            - name: APACHE_HTTP_PORT_NUMBER
              value: "8080"
            - name: APACHE_HTTPS_PORT_NUMBER
              value: "8443"
          image: docker.io/bitnami/wordpress:5.9.2-debian-10-r8
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 6
            httpGet:
              path: /wp-admin/install.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 120
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: wordpress
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
            - containerPort: 8443
              name: https
              protocol: TCP
          readinessProbe:
            failureThreshold: 6
            httpGet:
              path: /wp-login.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            requests:
              cpu: 300m
              memory: 512Mi
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
          volumeMounts:
            - mountPath: /bitnami/wordpress
              name: wordpress-data
              subPath: wordpress
      dnsPolicy: ClusterFirst
      hostAliases:
        - hostnames:
            - status.localhost
          ip: 127.0.0.1
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
        - name: wordpress-data
          persistentVolumeClaim:
            claimName: wordpress-ebs

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-13.1.3
  name: wordpress-ebs
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: gp2
  volumeMode: Filesystem
  volumeName: pvc-bf291331-af7c-47c6-983c-500c3aed734a
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi
  phase: Bound

with EFS (doesn't work as expected):

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-13.1.3
  name: wordpress-efs
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: wordpress-efs
      app.kubernetes.io/name: wordpress
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: wordpress-efs
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: wordpress
        helm.sh/chart: wordpress-13.1.3
    spec:
      affinity: {}
      containers:
        - env:
            - name: BITNAMI_DEBUG
              value: "true"
            - name: ALLOW_EMPTY_PASSWORD
              value: "yes"
            - name: MARIADB_HOST
              value: <EXTERNAL_DB_HOST>
            - name: MARIADB_PORT_NUMBER
              value: "3306"
            - name: WORDPRESS_DATABASE_NAME
              value: <EXTERNAL_DB_NAME>
            - name: WORDPRESS_DATABASE_USER
              value: <EXTERNAL_DB_USER>
            - name: WORDPRESS_DATABASE_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: mariadb-password
                  name: wordpress-efs-externaldb
            - name: WORDPRESS_USERNAME
              value: <WORDPRESS_USERNAME>
            - name: WORDPRESS_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: wordpress-password
                  name: wordpress-efs
            - name: WORDPRESS_EMAIL
              value: <WORDPRESS_EMAIL>
            - name: WORDPRESS_FIRST_NAME
              value: FirstName
            - name: WORDPRESS_LAST_NAME
              value: LastName
            - name: WORDPRESS_HTACCESS_OVERRIDE_NONE
              value: "no"
            - name: WORDPRESS_ENABLE_HTACCESS_PERSISTENCE
              value: "no"
            - name: WORDPRESS_BLOG_NAME
              value: User's Blog!
            - name: WORDPRESS_SKIP_BOOTSTRAP
              value: "no"
            - name: WORDPRESS_TABLE_PREFIX
              value: wp_
            - name: WORDPRESS_SCHEME
              value: http
            - name: WORDPRESS_EXTRA_WP_CONFIG_CONTENT
            - name: WORDPRESS_AUTO_UPDATE_LEVEL
              value: none
            - name: WORDPRESS_PLUGINS
              value: none
            - name: APACHE_HTTP_PORT_NUMBER
              value: "8080"
            - name: APACHE_HTTPS_PORT_NUMBER
              value: "8443"
          image: docker.io/bitnami/wordpress:5.9.2-debian-10-r8
          imagePullPolicy: IfNotPresent
          livenessProbe:
            failureThreshold: 6
            httpGet:
              path: /wp-admin/install.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 120
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          name: wordpress
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
            - containerPort: 8443
              name: https
              protocol: TCP
          readinessProbe:
            failureThreshold: 6
            httpGet:
              path: /wp-login.php
              port: http
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 5
          resources:
            requests:
              cpu: 300m
              memory: 512Mi
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
          volumeMounts:
            - mountPath: /bitnami/wordpress
              name: wordpress-data
              subPath: wordpress
      dnsPolicy: ClusterFirst
      hostAliases:
        - hostnames:
            - status.localhost
          ip: 127.0.0.1
      initContainers:
        - args:
            - -ec
            - |
              mkdir -p /bitnami/wordpress
              find /bitnami/wordpress -mindepth 0 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" | xargs -r chown -R 1001:1001
          command:
            - /bin/bash
          image: docker.io/bitnami/bitnami-shell:10-debian-10-r369
          imagePullPolicy: IfNotPresent
          name: volume-permissions
          resources: {}
          securityContext:
            runAsUser: 0
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /bitnami/wordpress
              name: wordpress-data
              subPath: wordpress
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1001
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
        - name: wordpress-data
          persistentVolumeClaim:
            claimName: wordpress-efs

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: efs.csi.aws.com
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: wordpress
    helm.sh/chart: wordpress-13.1.3
  name: wordpress-efs
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: efs-lyte-wordress-unencrypted
  volumeMode: Filesystem
  volumeName: pvc-155e9dc0-270c-4098-983e-fe7c935cbc93
status:
  accessModes:
    - ReadWriteMany
  capacity:
    storage: 10Gi
  phase: Bound

I also tested EFS with ReadWriteOnce access mode but it doesn't help.

After liveness probe failed and container became ready, some WordPress plugins does not work as expected (for example WP rocket). I read some related issues and possible solutions related to WP rocket plugin problem (for example this one), but in my case, I don't see any errors in WordPress admin console, and this plugin can write to necessary config files like .htaccess or wp-config.php. Anyway, even though the problem with WP rocket can be not related to the EFS issue, I'm not sure that WordPress is configured properly because of failures in preparation steps.

Thank you in advance!

migruiz4 commented 2 years ago

Hi @burakovsky,

I'm sorry but I may not be able to help you if it is an issue with EFS storage, since it would be out of my scope if it is not a problem with the Wordpress chart or image.

I have seen some users having similar issues with EFS when using non-root containers. My recommendation would be to try a simple test deployment, fine-tune EFS-related settings, then use it with the chart. Maybe the following helps you: https://devops.stackexchange.com/questions/13939/how-to-allow-a-non-root-user-to-write-to-a-mounted-efs-in-eks

If the problem persists, it would be helpful if you could log into the pod and check permissions under /bitnami/wordpress.

burakovsky commented 2 years ago

Hi @migruiz4,

Thank you for your responce! Yes, I checked the permissions and everything is ok but looks like efs performance is the main issue here. I started chart in diagnostic mode and ran entrypoint command manually - it finished successfully, but it took ~ 5 min to finish, so liveness probe just restarted a container earlier than all preparation steps finished.

Close this issue as there is no issues with Wordpress chart. Thank you!

devpro commented 1 year ago

Thank you so much @burakovsky. I was stuck with this issue. I'm running Kubernetes in Azure VMs with blob storage and I couldn't understand what was missing. The container takes more than 5 minutes to complete the installation of WordPress. You saved my day! I couldn't understand also why I got sometimes HTTP 503 (the wordpress service was missing because of the container issue).

lotyp commented 1 year ago

So, I have the same issue, and I'm using Amazon EFS driver. What helped is to enable startupProbe, which is disabled by default, and set the start time to around ~ 8 minutes.

startupProbe:
  enabled: true
  initialDelaySeconds: 660
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 9
  successThreshold: 1

The startup probe runs first and should be enabled. Liveness and Readiness should be set by default as is. I have tried with 16Gi volume.

bitnami / charts