[bitnami/ghost] Permission denied error on startup

Name and Version

bitnami/ghost:5.55.1-debian-11-r0

What architecture are you using?

None

What steps will reproduce the bug?

Running the latest bitnami/ghost helm chart at version 19.3.28

I have persistence.enabled: true and volumePermissions.enabled: true (although I'm not sure if that is necessary yet)

What is the expected behavior?

Pod starts up after configuration change.

What do you see instead?

Permission denied error for logging file at startup.

ghost 16:10:50.04 
ghost 16:10:50.05 Welcome to the Bitnami ghost container
ghost 16:10:50.05 Subscribe to project updates by watching https://github.com/bitnami/containers
ghost 16:10:50.05 Submit issues and feature requests at https://github.com/bitnami/containers/issues
ghost 16:10:50.05 
ghost 16:10:50.06 INFO  ==> Configuring libnss_wrapper
ghost 16:10:50.08 INFO  ==> Validating settings in MYSQL_CLIENT_* env vars
ghost 16:10:50.14 INFO  ==> Restoring persisted Ghost installation
ghost 16:10:50.17 INFO  ==> Trying to connect to the database server
ghost 16:10:50.47 INFO  ==> ** Ghost setup finished! **

ghost 16:10:50.50 INFO  ==> ** Starting Ghost **

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

- Inspecting operating system
[2023-07-26 16:10:52] INFO Ghost is running in production...
[2023-07-26 16:10:52] INFO Your site is now available on http://localhost:2368/
[2023-07-26 16:10:52] INFO Ctrl+C to shut down
[2023-07-26 16:10:52] INFO Ghost server started in 0.71s
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, open '/opt/bitnami/ghost/versions/5.55.1/content/logs/http___localhost_2368_production.error.log'
Emitted 'error' event on WriteStream instance at:
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/opt/bitnami/ghost/versions/5.55.1/content/logs/http___localhost_2368_production.error.log'
}

Additional information

I am running with GHOST_SKIP_BOOTSTRAP: yes, as I have already initialized a prior install in my external MySQL DB. I had been testing configuration updates to the ghost install to ensure it works properly but saw this error as I was running through those tests. It doesn't seem to be intermittent, and I am now stuck. A fresh helm install does not seem to resolve.

Something that seems a bit off to me is that when I look at the volume mounts in the running pod, I can see /bitnami/ghost as the mounted path, but in the above error, it seems to be trying for /opt/bitnami/ghost - Would this mismatch be related to an issue with permissions in the Dockerfile / user configuration?

    Mounts:
      /bitnami/ghost from ghost-data (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-kltpm (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  ghost-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  ghost
    ReadOnly:   false

I confirm this issue is still happening for helm chart for k8s as well, not only for docker.

The issue resides on a missing symlink at /opt/bitnami/ghost/content/logs (or under that), that makes it to arrive at /bitnami/ghost/contents/logs, since /bitnami/ghost/ is mounted at the PVC so it has write permissions.

More info, I was captured the configuration file that is built before it crashes, and I found that the logging parameter, at least for docker and helm charts are bad:

It shows

[...]
  },
  "logging": {
    "transports": [
      "file"
    ]
  },
  "process": "local",
  "paths": {
    "contentPath": "/opt/bitnami/ghost/content"
  }
}

When it should point it to the stdout, which is the only way you'll be able to get any logs at all on the docker/pod's logs handler:


  },
  "logging": {
    "transports": [
      "stdout"
    ]
  },
  "process": "local",
  "paths": {
    "contentPath": "/opt/bitnami/ghost/content"
  }
}

Hi @CeliaGMqrz ! What should be the most inline way to fix this from within the chart values so I don't have to rebuild the image? I saw that passing the logging as chart variable doesn't seem to produce any effects since it's not utilized when constructing the /opt/bitnami/ghost/config.production.json file.

Any ideas? :pray:

Okey, there seems to be more than just one problem here, above from the logging block pointing to file, which is not needed (or actually, harmful) for docker/kubernetes instead of pointing it to stdout, there appears to be issues with the PV mounting point as well.

I've tried to set the logging file predefined path to a temporal workdir using extraVolumes, and it is not failing to run that, but now it's failing to copy some files whose paths appears to be needed to be patched into pointing to the Persistant Volume, and in turn, it's trying to write into the RO part of the layers:

ghost 16:27:50.85 DEBUG ==> Getting database.connection.user from Ghost configuration
ghost 16:27:50.87 DEBUG ==> Getting database.connection.password from Ghost configuration
ghost 16:27:50.89 DEBUG ==> Executing SQL command:
SELECT 1
1

ghost 16:27:50.90 INFO  ==> ** Ghost setup finished! **
ghost 16:27:50.91 INFO  ==> ** Starting Ghost **

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

- Inspecting operating system
[2023-08-02 16:27:51] INFO Ghost is running in production...
[2023-08-02 16:27:51] INFO Your site is now available on http://localhost:2368/
[2023-08-02 16:27:51] INFO Ctrl+C to shut down
[2023-08-02 16:27:51] INFO Ghost server started in 0.404s
[2023-08-02 16:27:51] INFO Database is in a ready state.
[2023-08-02 16:27:51] INFO Ghost database ready in 0.588s
[2023-08-02 16:27:52] ERROR EACCES: permission denied, copyfile '/opt/bitnami/ghost/versions/5.57.1/core/server/services/route-settings/default-routes.yaml' -> '/opt/bitnami/ghost/versions/5.57.1/content/settings/routes.yaml'

EACCES: permission denied, copyfile '/opt/bitnami/ghost/versions/5.57.1/core/server/services/route-settings/default-routes.yaml' -> '/opt/bitnami/ghost/versions/5.57.1/content/settings/routes.yaml'

Error ID:
    87170990-3151-11ee-8867-7bb820e52112

Error Code: 
    EACCES

----------------------------------------

Error: EACCES: permission denied, copyfile '/opt/bitnami/ghost/versions/5.57.1/core/server/services/route-settings/default-routes.yaml' -> '/opt/bitnami/ghost/versions/5.57.1/content/settings/routes.yaml'
    at bootGhost (/opt/bitnami/ghost/versions/5.57.1/core/boot.js:555:32)

[2023-08-02 16:27:52] WARN Ghost is shutting down
[2023-08-02 16:27:52] WARN Ghost has shut down
[2023-08-02 16:27:52] WARN Your site is now offline
[2023-08-02 16:27:52] WARN Ghost was running for a few seconds

It seems that having the logging pointing to that path is just another result of not having the right directories mounted/pointing/symlinked to the PV, and that's why the whole thing fails.

:-(

Just for the record, as a temporal workaround, I've to set the chart to ghostSkipInstall: true so it does not attempt to write the configuration file, also defined extraEnvVars so that it does not write the logs into the filesystem, since it's not usable at all from either docker/kubernetes if its not sent to stdout.

extraEnvVars:
  - name: logging__transports
    value: '["stdout"]'

Hi @thatguyatgithub and @nickmonad

Just to double check. I think I could reproduce your issue following these steps:

# Install mysql
$ helm install mysql oci://registry-1.docker.io/bitnamicharts/mysql --version 9.x.x --set auth.rootPassword=root.12345 --set auth.database=bitnami_ghost --set auth.username=bn_ghost --set auth.password=ghost.12345
# Install ghost and initialize external database and PV
$ helm install ghost oci://registry-1.docker.io/bitnamicharts/ghost --set mysql.enabled=false --set externalDatabase.host=mysql --set externalDatabase.password=ghost.12345 --set persistence.enabled=true --set volumePermissions.enabled=true --set ghostHost=example.com --set ghostPassword=s3cr3tPa55
# Uninstall ghost. PV and PVC will remain in the cluster.
$ helm uninstall ghost
# Install ghost, reusing database and volumes
$ helm install ghost oci://registry-1.docker.io/bitnamicharts/ghost --set ghostSkipInstall=true --set mysql.enabled=false --set externalDatabase.host=mysql --set externalDatabase.password=ghost.12345 --set persistence.enabled=true --set volumePermissions.enabled=true --set ghostHost=example.com --set ghostPassword=s3cr3tPa55

The error I am facing is this one:

$ kubectl logs deploy/ghost
...
- Inspecting operating system
[2023-08-08 08:18:15] INFO Ghost is running in production...
[2023-08-08 08:18:15] INFO Your site is now available on http://localhost:2368/
[2023-08-08 08:18:15] INFO Ctrl+C to shut down
[2023-08-08 08:18:15] INFO Ghost server started in 0.531s
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, open '/opt/bitnami/ghost/versions/5.58.0/content/logs/http___localhost_2368_production.error.log'
Emitted 'error' event on WriteStream instance at:
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/opt/bitnami/ghost/versions/5.58.0/content/logs/http___localhost_2368_production.error.log'
}

Are we all on the same page?

@fmulero Yes, I believe so. Thanks for looking into it

Arg! the steps I shared are wrong. The error I was suffering appeared because the helm uninstall ghost removed the persistent volume claim and the volume in my case. In that case I was running the chart with a populated database and was empty volume.

Could you review if the volume is correctly attached? Could you try this command kubectl exec -it deploy/ghost -- ls -la in your deployment? You should see something like this:

$ kubectl exec -it deploy/ghost -- ls -la 
Defaulted container "ghost" out of: ghost, volume-permissions (init)
total 1820
drwxrwxrwx 1 ghost root    4096 Aug 10 16:32 .
drwxrwxr-x 1 root  root    4096 Aug  3 14:26 ..
-rw-rw-rw- 1 ghost root      83 Aug  3 12:33 .ghost-cli
-rw-r--r-- 1 root  root 1829875 Aug  3 12:35 .spdx-ghost.spdx
lrwxrwxrwx 1 root  root      52 Aug  3 12:33 bin -> /opt/bitnami/ghost/versions/5.57.3/node_modules/.bin
lrwxrwxrwx 1  1001 root      37 Aug 10 16:32 config.production.json -> /bitnami/ghost/config.production.json
lrwxrwxrwx 1  1001 root      22 Aug 10 16:32 content -> /bitnami/ghost/content
lrwxrwxrwx 1 root  root      34 Aug  3 12:33 current -> /opt/bitnami/ghost/versions/5.57.3
drwxr-xr-x 2 root  root    4096 Aug  3 12:56 licenses
drwxrwxr-x 2 root  root    4096 Aug  3 14:26 nss-wrapper
drwxr-xr-x 3 root  root    4096 Aug  3 12:56 versions

Facing the same issue

I am thinking maybe it is caused from this file? https://github.com/bitnami/containers/blob/main/bitnami/ghost/5/debian-11/rootfs/opt/bitnami/scripts/ghost/postunpack.sh

This is where the permissions are set. Maybe it's missing a folder?

Thanks @mike-callahan for sharing it.

Not sure if the problem is there, that script is executed at build time and the images seems to have the right permissions. I think there is a race condition which is broking the links under /opt/bitnami/ghost and likely due to permissions. Could you answer following questions or give us more information about your case?:

Are you using the helm chart? Could you share the values and the versions used?
Could you share the steps you followed to face this issue? are you upgrading from a previous version?
Could you run ls -la on /opt/bitnami/ghost and /bitnami/ghost?

Hi, we've just released a new chart version with the diagnosticMode feature. That could help us to diagnose the issue, you can enable it just setting the diagnosticMode.enabled value to true and the containers will be kept running, allowing you to check the folders and permissions.

Hi, Same use case. I have ran the last helm chart 19.4.1 to use diagnostic mode. Container security and fs group is activated into helm values. My database is external, and already initialized.

First start : diag mode false, Ghost startup failed with an file access error on :

│ stream logs failed container "ghost5" in pod "ghost5-54dd854b6c-g48jq" is waiting to start: ContainerCreating for ghost5/ghost5-54dd854b6c-g48jq (ghost5)                                                                                                                                      │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.42                                                                                                                                                                                                                                                      │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.42 Welcome to the Bitnami ghost container                                                                                                                                                                                                               │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.42 Subscribe to project updates by watching https://github.com/bitnami/containers                                                                                                                                                                       │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.43 Submit issues and feature requests at https://github.com/bitnami/containers/issues                                                                                                                                                                   │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.43                                                                                                                                                                                                                                                      │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.45 INFO  ==> Configuring libnss_wrapper                                                                                                                                                                                                                 │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.51 INFO  ==> Validating settings in MYSQL_CLIENT_* env vars                                                                                                                                                                                             │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.57 DEBUG ==> Validating settings in GHOST_* environment variables...                                                                                                                                                                                    │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.60 WARN  ==> The GHOST_SMTP_PASSWORD environment variable is empty or not set.                                                                                                                                                                          │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.65 INFO  ==> Restoring persisted Ghost installation                                                                                                                                                                                                     │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.73 INFO  ==> Trying to connect to the database server                                                                                                                                                                                                   │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.73 DEBUG ==> Getting database.connection.host from Ghost configuration                                                                                                                                                                                  │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.86 DEBUG ==> Getting database.connection.port from Ghost configuration                                                                                                                                                                                  │
│ ghost5-54dd854b6c-g48jq ghost 14:35:47.98 DEBUG ==> Getting database.connection.database from Ghost configuration                                                                                                                                                                              │
│ ghost5-54dd854b6c-g48jq ghost 14:35:48.10 DEBUG ==> Getting database.connection.user from Ghost configuration                                                                                                                                                                                  │
│ ghost5-54dd854b6c-g48jq ghost 14:35:48.23 DEBUG ==> Getting database.connection.password from Ghost configuration                                                                                                                                                                              │
│ ghost5-54dd854b6c-g48jq ghost 14:35:48.39 DEBUG ==> Executing SQL command:                                                                                                                                                                                                                     │
│ ghost5-54dd854b6c-g48jq SELECT 1                                                                                                                                                                                                                                                               │
│ ghost5-54dd854b6c-g48jq 1                                                                                                                                                                                                                                                                      │
│ ghost5-54dd854b6c-g48jq ghost 14:35:48.44 INFO  ==> ** Ghost setup finished! **                                                                                                                                                                                                                │
│ ghost5-54dd854b6c-g48jq                                                                                                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq ghost 14:35:48.51 INFO  ==> ** Starting Ghost **                                                                                                                                                                                                                       │
│ ghost5-54dd854b6c-g48jq                                                                                                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq https://careers.ghost.org                                                                                                                                                                                                                                              │
│ ghost5-54dd854b6c-g48jq                                                                                                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq - Inspecting operating system                                                                                                                                                                                                                                          │
│ ghost5-54dd854b6c-g48jq [2023-08-18 14:35:54] INFO Ghost is running in production...                                                                                                                                                                                                           │
│ ghost5-54dd854b6c-g48jq [2023-08-18 14:35:54] INFO Your site is now available on http://localhost:2368/                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq [2023-08-18 14:35:54] INFO Ctrl+C to shut down                                                                                                                                                                                                                         │
│ ghost5-54dd854b6c-g48jq [2023-08-18 14:35:54] INFO Ghost server started in 2.794s                                                                                                                                                                                                              │
│ ghost5-54dd854b6c-g48jq node:events:491                                                                                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq       throw er; // Unhandled 'error' event                                                                                                                                                                                                                             │
│ ghost5-54dd854b6c-g48jq       ^                                                                                                                                                                                                                                                                │
│ ghost5-54dd854b6c-g48jq                                                                                                                                                                                                                                                                        │
│ ghost5-54dd854b6c-g48jq Error: EACCES: permission denied, open '/opt/bitnami/ghost/versions/5.59.1/content/logs/http___localhost_2368_production.error.log'                                                                                                                                    │
│ ghost5-54dd854b6c-g48jq Emitted 'error' event on WriteStream instance at:                                                                                                                                                                                                                      │
│ ghost5-54dd854b6c-g48jq     at emitErrorNT (node:internal/streams/destroy:157:8)                                                                                                                                                                                                               │
│ ghost5-54dd854b6c-g48jq     at emitErrorCloseNT (node:internal/streams/destroy:122:3)                                                                                                                                                                                                          │
│ ghost5-54dd854b6c-g48jq     at processTicksAndRejections (node:internal/process/task_queues:83:21) {                                                                                                                                                                                           │
│ ghost5-54dd854b6c-g48jq   errno: -13,                                                                                                                                                                                                                                                          │
│ ghost5-54dd854b6c-g48jq   code: 'EACCES',                                                                                                                                                                                                                                                      │
│ ghost5-54dd854b6c-g48jq   syscall: 'open',                                                                                                                                                                                                                                                     │
│ ghost5-54dd854b6c-g48jq   path: '/opt/bitnami/ghost/versions/5.59.1/content/logs/http___localhost_2368_production.error.log'                                                                                                                                                                   │
│ ghost5-54dd854b6c-g48jq }                                                                                                                                                                                                                                                                      │
│ Stream closed EOF for ghost5/ghost5-54dd854b6c-g48jq (ghost5)                                                                                                                                                                                                                                  │
│

Second start : diag mode true, I launch the requested commands

ls -al /opt/bitnami/ghost
total 1784
drwxrwxrwx  6 ghost root    4096 Aug 15 14:21 .
drwxrwxr-x  9 root  root    4096 Aug 15 14:21 ..
-rw-rw-rw-  1 ghost root      83 Aug 15 12:22 .ghost-cli
-rw-r--r--  1 root  root 1794146 Aug 15 12:25 .spdx-ghost.spdx
lrwxrwxrwx  1 root  root      52 Aug 15 12:23 bin -> /opt/bitnami/ghost/versions/5.59.1/node_modules/.bin
drwxrwxrwx 11 ghost root    4096 Aug 15 13:03 content
lrwxrwxrwx  1 root  root      34 Aug 15 12:22 current -> /opt/bitnami/ghost/versions/5.59.1
drwxr-xr-x  2 root  root    4096 Aug 15 13:03 licenses
drwxrwxr-x  2 root  root    4096 Aug 15 14:21 nss-wrapper
drwxr-xr-x  3 root  root    4096 Aug 15 13:03 versions

ls -al /bitnami/ghost
total 32
drwxrwxrwx  4  1001  1001  4096 Aug 18 14:53 .
drwxr-xr-x  3 root  root   4096 Aug 15 14:23 ..
-rw-r--r--  1 ghost ghost   254 Aug 18 12:28 config.production.json
drwxrwxrwx 11 ghost ghost  4096 Aug 18 12:28 content
drwx------  2  1001  1001 16384 Aug 18 12:28 lost+found

ls -al /bitnami/ghost/content
total 44
drwxrwxrwx 11 ghost ghost 4096 Aug 18 12:28 .
drwxrwxrwx  4  1001  1001 4096 Aug 18 14:53 ..
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 apps
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 data
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 files
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 images
drwxrwxrwx  2 ghost ghost 4096 Aug 18 13:27 logs
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 media
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 public
drwxrwxrwx  2 ghost ghost 4096 Aug 18 12:28 settings
drwxrwxrwx  3 ghost ghost 4096 Aug 18 12:28 themes

ls -al /bitnami/ghost/content/logs
total 8
drwxrwxrwx  2 ghost ghost 4096 Aug 18 13:27 .
drwxrwxrwx 11 ghost ghost 4096 Aug 18 12:28 ..
-rw-r--r--  1 ghost ghost    0 Aug 18 13:27 http___localhost_2368_production.error.log
-rw-r--r--  1 ghost ghost    0 Aug 18 13:15 http___localhost_2368_production.log

To know the UID and GID of ghost user/group:

cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
ghost:x:999:999::/home/ghost:/bin/sh

I tried to modify my values like that:

    podSecurityContext:
      enabled: true
      fsGroup: 999
    containerSecurityContext:
      enabled: true
      runAsUser: 999
      runAsNonRoot: true

But the result is the same.

Thansk a lot @lcotonea.

The problem seems related with the creation of the symlink /opt/bitnami/ghost/content -> /bitnami/ghost/content.

Could you share also your values? I tried to reproduce the issue with this values.yaml file:

mysql:
  enabled: false
externalDatabase:
  host: mysql
  password: ghost.12345
persistence: 
  enabled: true
  existingClaim: ghost
volumePermissions:
  enabled: false
ghostHost: example.com
ghostPassword: s3cr3tPa55

And following these steps but I had no luck:

# Install mysql
$ helm install mysql oci://registry-1.docker.io/bitnamicharts/mysql --version 9.x.x --set auth.rootPassword=root.12345 --set auth.database=bitnami_ghost --set auth.username=bn_ghost --set auth.password=ghost.12345
# Create PVC (It doesn't have any relevant configuration)
$ kubectl apply -f $HOME/workspace/support/charts/18031/pvc.yaml
# Install ghost and initialize external database and PV
$ helm install ghost oci://registry-1.docker.io/bitnamicharts/ghost --version 19.3.34 -f values.yaml
# At this point I played a little with the app to generate logs and content.
# Uninstall ghost. PV and PVC will remain in the cluster.
$ helm uninstall ghost
$ helm install ghost oci://registry-1.docker.io/bitnamicharts/ghost --set image.debug=true --set ghostSkipInstall=true -f values.yaml

Hi @fmulero ,

I'm agree with your analysis about the root cause.

Here my value file : https://gist.github.com/lcotonea/996740b57119a02b7dd0e3882e77791e

Moreover, I check my config.production.json: It only contains the database settings

Then, I have made this test :

remove Helm release
remove PVC
create new volume
recreate PVC using the new volume
database's tables removal
helm release deployment (with same values, but with the "ghostSkipInstall" value to false

Everything is fine. I have checked my config.production.json and it is complete (database, log, server, url, etc.).

Maybe something wrong during helm update (day 2) that drives to inconsistent config file??

Hi @lcotonea

I was digging into the code and I could reproduce the error only forcing it. I mean, forcing a misalignment between the database and the persistent storage, for example using an existing database with a clean persistent volume (or a volume without the expected contents). Could your volumes be removed? Could you check the persistentVolumeReclaimPolicy in your volumes?

At the moment I don't find any other explanation.

I agree with @lcotonea https://github.com/bitnami/charts/issues/18031#issuecomment-1687712198. Based on my experience, I've guessed that this issue is caused by inconsistencies between the data in the MySQL database and the configuration data generated during the Ghost installation process.

If you're using an external database or persistent internal database storage, and you've tried uninstalling and reinstalling Ghost, this problem may arise. My assumption is that the existing data in the database doesn't match the newly generated configuration data during the Ghost pod installation process.

Therefore,

If you're using an external database, make sure there's no data left from the previous installation process in the database before reinstalling Ghost.
If you're using Helm to install MySQL as a dependency and plan to perform upgrades later, ensure that data persistence is configured for both MySQL and Ghost(in values.yaml, persistence.enabled=true, mysql.primary.persistence.enabled=true). This way, during a helm upgrade, the installation process can be skipped(ghostSkipInstall: false), and everything should work as expected.

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

I'm getting the same issue.

Hi @deba10106

Could you define "same issue"? What were the steps you followed to face it? Did you reinstall the chart? Were the Ghost or MySQL persistent volumes removed? Have you seen this comment?

I have a similar issue now when running ghost update on my fresh Lightsail Bitnami Ghost instance.

Message: The directory /opt/bitnami/ghost is not writable by your user. You must grant write access and try again.

And then when trying to do this, hitting bitnami:~/stack/ghost$ setfacl -m u:bitnami:rwx /opt/bitnami/ghost setfacl: /opt/bitnami/ghost: Operation not permitted

What do I do wrong?

EDIT: I tried to manually fix the permissions via sudo chmod -R 777 ghost but now ghost just crashes on update or backup. Now getting "✖ Stopping Ghost. A CliError occurred." Message: An unexpected error occurred while stopping Ghost.

I cannot even call ghost stop, it gives the same error. Rebooting my instance didn't help.

According to ghost documentation:

Switch to the user you used to setup your Ghost install for running Ghost commands

In your case you have to run the commands with the ghost user.

If you have further questions about the Ghost VM, please open a new issue in bitnami/vms

Facing this same issue, while doing a fresh deployment and using MYSQL as dependency.

Hi @druheendas

Could you define "same issue"? What were the steps you followed to face it? Did you reinstall the chart? Were the Ghost or MySQL persistent volumes removed? Have you seen this https://github.com/bitnami/charts/issues/18031#issuecomment-1698836964?. Please open a new issue specifying:

Chart version
Configuration
Steps to reproduce it.

I am running helm chart version 19.8.7. I have tried multiple combinations with the values files. Each time it failed I deleted the entire Namespace and made sure the PV's are also deleted. I am using the dependent MySql chart.

First start : diag mode false, debug mode true, volumePermissions.enabled:true, ghostHost: "blog.beta.xxxxxxxxx.com" Ghost startup failed with an file access error on :

ghost-blog ghost 13:26:03.48 INFO  ==> 
ghost-blog ghost 13:26:03.48 INFO  ==> Welcome to the Bitnami ghost container
ghost-blog ghost 13:26:03.48 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
ghost-blog ghost 13:26:03.49 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
ghost-blog ghost 13:26:03.49 INFO  ==> 
ghost-blog ghost 13:26:03.49 INFO  ==> Configuring libnss_wrapper
ghost-blog ghost 13:26:03.54 INFO  ==> Validating settings in MYSQL_CLIENT_* env vars
ghost-blog ghost 13:26:03.57 DEBUG ==> Validating settings in GHOST_* environment variables...
ghost-blog ghost 13:26:03.58 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
ghost-blog ghost 13:26:03.64 INFO  ==> Ensuring Ghost directories exist
ghost-blog ghost 13:26:03.64 INFO  ==> Trying to connect to the database server
ghost-blog ghost 13:26:03.65 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog ERROR 2002 (HY000): Can't connect to server on 'ghost-blog-mysql' (115)
ghost-blog ghost 13:26:08.66 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog ERROR 2002 (HY000): Can't connect to server on 'ghost-blog-mysql' (115)
ghost-blog ghost 13:26:13.67 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog ERROR 2002 (HY000): Can't connect to server on 'ghost-blog-mysql' (115)
ghost-blog ghost 13:26:18.69 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog ERROR 2002 (HY000): Can't connect to server on 'ghost-blog-mysql' (115)
ghost-blog ghost 13:26:23.70 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog ERROR 2002 (HY000): Can't connect to server on 'ghost-blog-mysql' (115)
ghost-blog ghost 13:26:28.72 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog 1
ghost-blog ghost 13:26:28.74 INFO  ==> Configuring database
ghost-blog ghost 13:26:28.78 INFO  ==> Setting up Ghost
ghost-blog 
ghost-blog Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
ghost-blog https://careers.ghost.org
ghost-blog 
ghost-blog - Inspecting operating system
ghost-blog - Checking for deprecations
ghost-blog 
ghost-blog [13:26:31] Checking for Ghost-CLI updates [started]
ghost-blog [13:26:31] Checking for Ghost-CLI updates [completed]
ghost-blog 
ghost-blog [13:26:31] Configuring Ghost [started]
ghost-blog [13:26:31] Configuring Ghost [completed]
ghost-blog [13:26:31] Setting up instance [started]
ghost-blog [13:26:31] Setting up instance [completed]
ghost-blog 
ghost-blog Ghost uses direct mail by default. To set up an alternative email method read our docs at https://ghost.org/docs/config/#mail
ghost-blog 
ghost-blog ------------------------------------------------------------------------------
ghost-blog 
ghost-blog Ghost was installed successfully! To complete setup of your publication, visit: 
ghost-blog 
ghost-blog     http://blog.beta.xxxxxxx.com/*/ghost/
ghost-blog 
ghost-blog ghost 13:26:31.67 INFO  ==> Configuring Ghost URL to http://blog.beta.xxxxxxx.com/*
ghost-blog ghost 13:26:31.77 INFO  ==> Passing admin user creation wizard
ghost-blog ghost 13:26:31.78 INFO  ==> Starting Ghost in background
ghost-blog ghost 13:27:05.65 DEBUG ==> Checking that /opt/bitnami/ghost/content/logs/ghost.log log file contains entry "Your admin interface is located at"
ghost-blog ghost 13:27:05.65 DEBUG ==> Found entry "Your admin interface is located at" in /opt/bitnami/ghost/content/logs/ghost.log
Stream closed EOF for ghost/ghost-blog-d98597ddd-qp4vq (volume-permissions)
ghost-blog                     <span class="gh-card-author">By Ghost</span>
ghost-blog                     <time class="gh-card-date" datetime="2024-02-12">Feb 12, 2024</time>
ghost-blog                 <!--
ghost-blog          --></footer>
ghost-blog         </div>
ghost-blog     </a>
ghost-blog </article>                    
ghost-blog 
ghost-blog 
ghost-blog             </div>
ghost-blog         </main>
ghost-blog 
ghost-blog 
ghost-blog 
ghost-blog     </div>
ghost-blog </section>
ghost-blog     
ghost-blog     <footer class="gh-footer gh-outer">
ghost-blog     <div class="gh-footer-inner gh-inner">
ghost-blog 
ghost-blog         <div class="gh-footer-bar">
ghost-blog             <span class="gh-footer-logo is-title">
ghost-blog                     Ghost
ghost-blog             </span>
ghost-blog             <nav class="gh-footer-menu">
ghost-blog                 <ul class="nav">
ghost-blog     <li class="nav-sign-up nav-current"><a href="#/portal/">Sign up</a></li>
ghost-blog </ul>
ghost-blog 
ghost-blog             </nav>
ghost-blog             <div class="gh-footer-copyright">
ghost-blog                 Powered by <a href="https://ghost.org/" target="_blank" rel="noopener">Ghost</a>
ghost-blog             </div>
ghost-blog         </div>
ghost-blog         
ghost-blog                 <section class="gh-footer-signup">
ghost-blog                     <h2 class="gh-footer-signup-header is-title">
ghost-blog                         Ghost
ghost-blog                     </h2>
ghost-blog                     <p class="gh-footer-signup-subhead is-body">
ghost-blog                         Thoughts, stories and ideas
ghost-blog                     </p>
ghost-blog                     <form class="gh-form" data-members-form>
ghost-blog     <input class="gh-form-input" type="email" placeholder="jamie@example.com" required data-members-email>
ghost-blog     <button class="gh-button" type="submit">
ghost-blog         <span><span>Subscribe</span> <svg xmlns="http://www.w3.org/2000/svg" width="32" height="32" fill="currentColor" viewBox="0 0 256 256"><path d="M224.49,136.49l-72,72a12,12,0,0,1-17-17L187,140H40a12,12,0,0,1,0-24H187L135.51,64.48a12,12,0,0,1,17-17l72,72A12,12,0,0,1,224.49,136.49Z"></path></svg></span>
ghost-blog         <svg xmlns="http://www.w3.org/2000/svg" height="24" width="24" viewBox="0 0 24 24">
ghost-blog     <g stroke-linecap="round" stroke-width="2" fill="currentColor" stroke="none" stroke-linejoin="round" class="nc-icon-wrapper">
ghost-blog         <g class="nc-loop-dots-4-24-icon-o">
ghost-blog             <circle cx="4" cy="12" r="3"></circle>
ghost-blog             <circle cx="12" cy="12" r="3"></circle>
ghost-blog             <circle cx="20" cy="12" r="3"></circle>
ghost-blog         </g>
ghost-blog         <style data-cap="butt">
ghost-blog             .nc-loop-dots-4-24-icon-o{--animation-duration:0.8s}
ghost-blog             .nc-loop-dots-4-24-icon-o *{opacity:.4;transform:scale(.75);animation:nc-loop-dots-4-anim var(--animation-duration) infinite}
ghost-blog             .nc-loop-dots-4-24-icon-o :nth-child(1){transform-origin:4px 12px;animation-delay:-.3s;animation-delay:calc(var(--animation-duration)/-2.666)}
ghost-blog             .nc-loop-dots-4-24-icon-o :nth-child(2){transform-origin:12px 12px;animation-delay:-.15s;animation-delay:calc(var(--animation-duration)/-5.333)}
ghost-blog             .nc-loop-dots-4-24-icon-o :nth-child(3){transform-origin:20px 12px}
ghost-blog             @keyframes nc-loop-dots-4-anim{0%,100%{opacity:.4;transform:scale(.75)}50%{opacity:1;transform:scale(1)}}
ghost-blog         </style>
ghost-blog     </g>
ghost-blog </svg>        <svg class="checkmark" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 52 52">
ghost-blog     <path class="checkmark__check" fill="none" d="M14.1 27.2l7.1 7.2 16.7-16.8"/>
ghost-blog     <style>
ghost-blog         .checkmark {
ghost-blog             width: 40px;
ghost-blog             height: 40px;
ghost-blog             display: block;
ghost-blog             stroke-width: 2.5;
ghost-blog             stroke: currentColor;
ghost-blog             stroke-miterlimit: 10;
ghost-blog         }
ghost-blog 
ghost-blog         .checkmark__check {
ghost-blog             transform-origin: 50% 50%;
ghost-blog             stroke-dasharray: 48;
ghost-blog             stroke-dashoffset: 48;
ghost-blog             animation: stroke .3s cubic-bezier(0.650, 0.000, 0.450, 1.000) forwards;
ghost-blog         }
ghost-blog 
ghost-blog         @keyframes stroke {
ghost-blog             100% { stroke-dashoffset: 0; }
ghost-blog         }
ghost-blog     </style>
ghost-blog </svg>    </button>
ghost-blog </form>                </section>
ghost-blog 
ghost-blog     </div>
ghost-blog </footer>    
ghost-blog </div>
ghost-blog 
ghost-blog 
ghost-blog <script src="***/*/assets/built/source.js?v=244a480aae"></script>
ghost-blog 
ghost-blog 
ghost-blog 
ghost-blog </body>
ghost-blog </html>
ghost-blog ghost 13:27:10.77 ERROR ==> An error occurred while installing Ghost
Stream closed EOF for ghost/ghost-blog-d98597ddd-qp4vq (ghost-blog)

Second start : diag mode false, debug mode true, volumePermissions.enabled:true, ghostHost: "blog.beta.xxxxxxxxx.com" This time did not terminate the mysql, deleted the deployment of blog and ran helm upgrade with ghostSkipInstall:true, Ghost startup failed with an file access error on :

ghost-blog ghost 14:09:41.09 INFO  ==> 
ghost-blog ghost 14:09:41.09 INFO  ==> Welcome to the Bitnami ghost container
ghost-blog ghost 14:09:41.09 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
ghost-blog ghost 14:09:41.09 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
ghost-blog ghost 14:09:41.09 INFO  ==> 
ghost-blog ghost 14:09:41.10 INFO  ==> Configuring libnss_wrapper
ghost-blog ghost 14:09:41.14 INFO  ==> Validating settings in MYSQL_CLIENT_* env vars
ghost-blog ghost 14:09:41.18 DEBUG ==> Validating settings in GHOST_* environment variables...
ghost-blog ghost 14:09:41.24 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
ghost-blog ghost 14:09:41.26 INFO  ==> Restoring persisted Ghost installation
ghost-blog ghost 14:09:41.34 INFO  ==> Trying to connect to the database server
ghost-blog ghost 14:09:41.34 DEBUG ==> Getting database.connection.host from Ghost configuration
ghost-blog ghost 14:09:41.38 DEBUG ==> Getting database.connection.port from Ghost configuration
ghost-blog ghost 14:09:41.47 DEBUG ==> Getting database.connection.database from Ghost configuration
ghost-blog ghost 14:09:41.56 DEBUG ==> Getting database.connection.user from Ghost configuration
ghost-blog ghost 14:09:41.66 DEBUG ==> Getting database.connection.password from Ghost configuration
ghost-blog ghost 14:09:41.76 DEBUG ==> Executing SQL command:
ghost-blog SELECT 1
ghost-blog 1
ghost-blog ghost 14:09:41.78 INFO  ==> ** Ghost setup finished! **
ghost-blog 
ghost-blog ghost 14:09:41.84 INFO  ==> ** Starting Ghost **
Stream closed EOF for ghost/ghost-blog-c777677d9-kglq7 (volume-permissions)
ghost-blog 
ghost-blog Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
ghost-blog https://careers.ghost.org
ghost-blog 
ghost-blog - Inspecting operating system
ghost-blog [2024-02-12 14:09:46] INFO Ghost is running in production...
ghost-blog [2024-02-12 14:09:46] INFO Your site is now available on http://localhost:2368/
ghost-blog [2024-02-12 14:09:46] INFO Ctrl+C to shut down
ghost-blog [2024-02-12 14:09:46] INFO Ghost server started in 2.124s
ghost-blog node:events:495
ghost-blog       throw er; // Unhandled 'error' event
ghost-blog       ^
ghost-blog 
ghost-blog Error: EACCES: permission denied, open '/opt/bitnami/ghost/versions/5.79.1/content/logs/http___localhost_2368_production.error.log'
ghost-blog Emitted 'error' event on WriteStream instance at:
ghost-blog     at emitErrorNT (node:internal/streams/destroy:151:8)
ghost-blog     at emitErrorCloseNT (node:internal/streams/destroy:116:3)
ghost-blog     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
ghost-blog   errno: -13,
ghost-blog   code: 'EACCES',
ghost-blog   syscall: 'open',
ghost-blog   path: '/opt/bitnami/ghost/versions/5.79.1/content/logs/http___localhost_2368_production.error.log'
ghost-blog }
ghost-blog 
ghost-blog Node.js v18.19.0
Stream closed EOF for ghost/ghost-blog-c777677d9-kglq7 (ghost-blog)

I have tried with volumePermissions.enabled:false also tried changing podSecurityContext.fsGroup & containerSecurityContext.runAsUser to 999 for ghost, 0 for root but the same error comes up.

Third start : diag mode true, I launch the requested commands

I have no name!@ghost-blog-bbbb587d6-79nkq:/opt/bitnami/ghost$ ls -al /opt/bitnami/ghost
total 3276
drwxrwxrwx  6 ghost root     140 Feb  9 20:15 .
drwxrwxr-x  9 root  root     135 Feb  9 20:15 ..
-rw-rw-rw-  1 ghost root      83 Feb  9 18:51 .ghost-cli
-rw-r--r--  1 root  root 3350036 Feb  9 19:49 .spdx-ghost.spdx
lrwxrwxrwx  1 root  root      52 Feb  9 18:51 bin -> /opt/bitnami/ghost/versions/5.79.1/node_modules/.bin
drwxrwxrwx 11 ghost root     126 Feb  9 20:15 content
lrwxrwxrwx  1 root  root      34 Feb  9 18:51 current -> /opt/bitnami/ghost/versions/5.79.1
drwxr-xr-x  2 root  root      30 Feb  9 19:06 licenses
drwxrwxr-x  2 root  root      41 Feb  9 20:13 nss-wrapper
drwxr-xr-x  3 root  root      20 Feb  9 20:15 versions

I have no name!@ghost-blog-bbbb587d6-79nkq:/opt/bitnami/ghost$ ls -al /bitnami/ghost
total 28
drwxrwsr-x  4 1001 1001  4096 Feb 12 14:09 .
drwxr-xr-x  3 root root    19 Feb  9 20:15 ..
-rw-rw-r--  1 1001 1001   239 Feb 12 14:09 config.production.json
drwxrwsrwx 11 1001 1001  4096 Feb 12 14:09 content
drwxrws---  2 1001 1001 16384 Feb 12 14:07 lost+found

I have no name!@ghost-blog-bbbb587d6-79nkq:/opt/bitnami/ghost$ ls -al /bitnami/ghost/content
total 44
drwxrwsrwx 11 1001 1001 4096 Feb 12 14:09 .
drwxrwsr-x  4 1001 1001 4096 Feb 12 14:09 ..
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 apps
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 data
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 files
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 images
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 logs
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 media
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 public
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 settings
drwxrwsrwx  4 1001 1001 4096 Feb 12 14:09 themes

I have no name!@ghost-blog-bbbb587d6-79nkq:/opt/bitnami/ghost$ ls -al /bitnami/ghost/content/logs
total 8
drwxrwsrwx  2 1001 1001 4096 Feb 12 14:09 .
drwxrwsrwx 11 1001 1001 4096 Feb 12 14:09 ..

I have no name!@ghost-blog-bbbb587d6-79nkq:/opt/bitnami/ghost$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
ghost:x:999:999::/home/ghost:/bin/sh

Fourth start : diag mode true, fresh start after deleting all PV and namespace with the following values.yaml file. I launch the requested commands:

values.yaml file

## Enable diagnostic mode in the deployment
##
diagnosticMode:
  ## @param diagnosticMode.enabled Enable diagnostic mode (all probes will be disabled and the command will be overridden)
  ##
  enabled: true
  ## @param diagnosticMode.command Command to override all containers in the deployment
  ##
  command:
    - sleep
  ## @param diagnosticMode.args Args to override all containers in the deployment
  ##
  args:
    - infinity

ghostHost: "blog.beta.xxxxxxx.com"
ghostSkipInstall: false

nodeSelector:
  role: infra

tolerations:
  - key: "node"
    operator: "Equal"
    value: "infra"
    effect: "NoSchedule"

resources:
  requests:
    cpu: "400m"
    memory: "1500M"
  limits:
    cpu: "500m"
    memory: "2G"

service:
  type: ClusterIP

ingress:
  enabled: true
  hostname: blog.beta.xxxxxxx.com
  path: /*
  annotations:
    alb.ingress.kubernetes.io/backend-protocol-version: HTTP1
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-south-1:111111111111:certificate/xxxx-xxxxxx-xxxxx-xxxxxx-xxxxx
    alb.ingress.kubernetes.io/group.name: beta-alb-internal
    alb.ingress.kubernetes.io/load-balancer-name: beta-alb-internal
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: '60'
    alb.ingress.kubernetes.io/healthcheck-timeout-seconds: '10'
    alb.ingress.kubernetes.io/healthcheck-path: /
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP": 80}]'
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/target-type: ip
    external-dns.alpha.kubernetes.io/dns-type: private

  ingressClassName: alb

persistence:
  enabled: true
  accessModes:
    - ReadWriteOnce
  size: 8Gi

volumePermissions:
  enabled: true

mysql:
  enabled: true
  architecture: standalone
  auth:
    rootPassword: ""
    database: bitnami_ghost
    username: bn_ghost
    password: ""
    existingSecret: ""
  primary:
    nodeSelector:
      role: infra
    tolerations:
      - key: "node"
        operator: "Equal"
        value: "infra"
        effect: "NoSchedule"
    persistence:
      enabled: true
      storageClass: ""
      accessModes:
        - ReadWriteOnce
      size: 8Gi

I have no name!@ghost-blog-5b9c5cdcc4-8rm98:/opt/bitnami/ghost$ cd ../scripts/ghost
I have no name!@ghost-blog-5b9c5cdcc4-8rm98:/opt/bitnami/scripts/ghost$ ./entrypoint.sh ./run.sh
ghost 14:42:19.57 INFO  ==>
ghost 14:42:19.57 INFO  ==> Welcome to the Bitnami ghost container
ghost 14:42:19.57 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
ghost 14:42:19.57 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
ghost 14:42:19.57 INFO  ==>
ghost 14:42:19.58 INFO  ==> Configuring libnss_wrapper

ghost 14:42:19.61 INFO  ==> ** Starting Ghost **

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

The `ghost run` command is used by the configured Ghost process manager and for debugging. If you're not running this to debug something, you should run `ghost start` instead.
[2024-02-12 14:42:24] INFO Ghost is running in production...
[2024-02-12 14:42:24] INFO Your site is now available on http://localhost:2368/
[2024-02-12 14:42:24] INFO Ctrl+C to shut down
[2024-02-12 14:42:24] INFO Ghost server started in 2.129s
node:events:495
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, open '/opt/bitnami/ghost/versions/5.79.1/content/logs/http___localhost_2368_production.error.log'
Emitted 'error' event on WriteStream instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/opt/bitnami/ghost/versions/5.79.1/content/logs/http___localhost_2368_production.error.log'
}

Node.js v18.19.0

I have no name!@ghost-blog-5b9c5cdcc4-8rm98:/opt/bitnami/scripts/ghost$ ls -al /opt/bitnami/ghost
total 3276
drwxrwxrwx  6 ghost root     140 Feb  9 20:15 .
drwxrwxr-x  9 root  root     135 Feb  9 20:15 ..
-rw-rw-rw-  1 ghost root      83 Feb  9 18:51 .ghost-cli
-rw-r--r--  1 root  root 3350036 Feb  9 19:49 .spdx-ghost.spdx
lrwxrwxrwx  1 root  root      52 Feb  9 18:51 bin -> /opt/bitnami/ghost/versions/5.79.1/node_modules/.bin
drwxrwxrwx 11 ghost root     126 Feb  9 20:15 content
lrwxrwxrwx  1 root  root      34 Feb  9 18:51 current -> /opt/bitnami/ghost/versions/5.79.1
drwxr-xr-x  2 root  root      30 Feb  9 19:06 licenses
drwxrwxr-x  2 root  root      41 Feb  9 20:13 nss-wrapper
drwxr-xr-x  3 root  root      20 Feb  9 20:15 versions

I have no name!@ghost-blog-5b9c5cdcc4-8rm98:/opt/bitnami/scripts/ghost$ ls -al /bitnami/ghost
total 20
drwxrwsr-x 3 1001 1001  4096 Feb 12 14:41 .
drwxr-xr-x 3 root root    19 Feb  9 20:15 ..
drwxrws--- 2 1001 1001 16384 Feb 12 14:41 lost+found

I have no name!@ghost-blog-5b9c5cdcc4-8rm98:/opt/bitnami/scripts/ghost$ cat /etc/passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
_apt:x:100:65534::/nonexistent:/usr/sbin/nologin
ghost:x:999:999::/home/ghost:/bin/sh

Fifth start : diag mode false, with above values.yaml file, then deleted the blog deployment and then again ran helm upgrade with ghostSkipInstall:true, diagnosticMode.enabled:true. I launch the requested commands:

I have no name!@ghost-blog-7cdf7f8967-ggbcb:/opt/bitnami/ghost$ find -L / -name "config.production.json"
/opt/bitnami/ghost/versions/5.79.1/core/shared/config/env/config.production.json
/opt/bitnami/ghost/current/core/shared/config/env/config.production.json
/bitnami/ghost/config.production.json
find: '/etc/ssl/private': Permission denied
find: '/usr/lib/ssl/private': Permission denied
find: '/var/cache/ldconfig': Permission denied
find: '/root': Permission denied
find: '/proc/tty/driver': Permission denied
/proc/self/task/14/fd/3/versions/5.79.1/core/shared/config/env/config.production.json
/proc/self/task/14/fd/3/current/core/shared/config/env/config.production.json
find: '/proc/self/task/14/fd/4': No such file or directory
/proc/self/task/14/cwd/versions/5.79.1/core/shared/config/env/config.production.json
/proc/self/task/14/cwd/current/core/shared/config/env/config.production.json
^C
I have no name!@ghost-blog-7cdf7f8967-ggbcb:/opt/bitnami/ghost$ cat /opt/bitnami/ghost/versions/5.79.1/core/shared/config/env/config.production.json
{
    "database": {
        "client": "mysql",
        "connection": {
            "host"     : "127.0.0.1",
            "user"     : "root",
            "password" : "",
            "database" : "ghost"
        }
    },
    "paths": {
        "contentPath": "content/"
    },
    "logging": {
        "level": "info",
        "rotation": {
            "enabled": true
        },
        "transports": ["file", "stdout"]
    }
}

I have no name!@ghost-blog-7cdf7f8967-ggbcb:/opt/bitnami/scripts$ cat /opt/bitnami/scripts/ghost-env.sh
#!/bin/bash
# Copyright VMware, Inc.
# SPDX-License-Identifier: APACHE-2.0
#
# Environment configuration for ghost

# The values for all environment variables will be set in the below order of precedence
# 1. Custom environment variables defined below after Bitnami defaults
# 2. Constants defined in this file (environment variables with no default), i.e. BITNAMI_ROOT_DIR
# 3. Environment variables overridden via external files using *_FILE variables (see below)
# 4. Environment variables set externally (i.e. current Bash context/Dockerfile/userdata)

# Load logging library
# shellcheck disable=SC1090,SC1091
. /opt/bitnami/scripts/liblog.sh

export BITNAMI_ROOT_DIR="/opt/bitnami"
export BITNAMI_VOLUME_DIR="/bitnami"

# Logging configuration
export MODULE="${MODULE:-ghost}"
export BITNAMI_DEBUG="${BITNAMI_DEBUG:-false}"

# By setting an environment variable matching *_FILE to a file path, the prefixed environment
# variable will be overridden with the value specified in that file
ghost_env_vars=(
    GHOST_DATA_TO_PERSIST
    GHOST_ENABLE_HTTPS
    GHOST_EXTERNAL_HTTP_PORT_NUMBER
    GHOST_EXTERNAL_HTTPS_PORT_NUMBER
    GHOST_HOST
    GHOST_PORT_NUMBER
    GHOST_BLOG_TITLE
    GHOST_SKIP_BOOTSTRAP
    GHOST_USERNAME
    GHOST_PASSWORD
    GHOST_EMAIL
    GHOST_SMTP_HOST
    GHOST_SMTP_PORT_NUMBER
    GHOST_SMTP_USER
    GHOST_SMTP_PASSWORD
    GHOST_SMTP_PROTOCOL
    GHOST_DATABASE_HOST
    GHOST_DATABASE_PORT_NUMBER
    GHOST_DATABASE_NAME
    GHOST_DATABASE_USER
    GHOST_DATABASE_PASSWORD
    GHOST_DATABASE_ENABLE_SSL
    GHOST_DATABASE_SSL_CA_FILE
    BLOG_TITLE
    SMTP_HOST
    SMTP_PORT
    GHOST_SMTP_PORT
    SMTP_USER
    SMTP_PASSWORD
    SMTP_PROTOCOL
    MYSQL_HOST
    MYSQL_PORT_NUMBER
    MYSQL_DATABASE_NAME
    MYSQL_DATABASE_USER
    MYSQL_DATABASE_PASSWORD
)
for env_var in "${ghost_env_vars[@]}"; do
    file_env_var="${env_var}_FILE"
    if [[ -n "${!file_env_var:-}" ]]; then
        if [[ -r "${!file_env_var:-}" ]]; then
            export "${env_var}=$(< "${!file_env_var}")"
            unset "${file_env_var}"
        else
            warn "Skipping export of '${env_var}'. '${!file_env_var:-}' is not readable."
        fi
    fi
done
unset ghost_env_vars

# Paths
export GHOST_BASE_DIR="${BITNAMI_ROOT_DIR}/ghost"
export GHOST_BIN_DIR="${GHOST_BASE_DIR}/bin"
export GHOST_LOG_FILE="${GHOST_BASE_DIR}/content/logs/ghost.log"
export GHOST_CONF_FILE="${GHOST_BASE_DIR}/config.production.json"
export GHOST_PID_FILE="${GHOST_BASE_DIR}/.ghostpid"
export PATH="${GHOST_BIN_DIR}:${BITNAMI_ROOT_DIR}/common/bin:${BITNAMI_ROOT_DIR}/node/bin:${PATH}"

# Ghost persistence configuration
export GHOST_VOLUME_DIR="${BITNAMI_VOLUME_DIR}/ghost"
export GHOST_DATA_TO_PERSIST="${GHOST_DATA_TO_PERSIST:-content config.production.json}"

# System users (when running with a privileged user)
export GHOST_DAEMON_USER="ghost"
export GHOST_DAEMON_GROUP="ghost"

# Ghost configuration
export GHOST_ENABLE_HTTPS="${GHOST_ENABLE_HTTPS:-no}" # only used during the first initialization
export GHOST_EXTERNAL_HTTP_PORT_NUMBER="${GHOST_EXTERNAL_HTTP_PORT_NUMBER:-80}" # only used during the first initialization
export GHOST_EXTERNAL_HTTPS_PORT_NUMBER="${GHOST_EXTERNAL_HTTPS_PORT_NUMBER:-443}" # only used during the first initialization
export GHOST_HOST="${GHOST_HOST:-localhost}" # only used during the first initialization
export GHOST_DEFAULT_PORT_NUMBER="2368" # only used at build time
export GHOST_PORT_NUMBER="${GHOST_PORT_NUMBER:-}" # only used during the first initialization
GHOST_BLOG_TITLE="${GHOST_BLOG_TITLE:-"${BLOG_TITLE:-}"}"
export GHOST_BLOG_TITLE="${GHOST_BLOG_TITLE:-"User's blog"}" # only used during the first initialization
export GHOST_SKIP_BOOTSTRAP="${GHOST_SKIP_BOOTSTRAP:-}" # only used during the first initialization

# Ghost credentials
export GHOST_USERNAME="${GHOST_USERNAME:-user}" # only used during the first initialization
export GHOST_PASSWORD="${GHOST_PASSWORD:-bitnami123}" # only used during the first initialization
export GHOST_EMAIL="${GHOST_EMAIL:-user@example.com}" # only used during the first initialization

# Ghost SMTP credentials
GHOST_SMTP_HOST="${GHOST_SMTP_HOST:-"${SMTP_HOST:-}"}"
export GHOST_SMTP_HOST="${GHOST_SMTP_HOST:-}" # only used during the first initialization
GHOST_SMTP_PORT_NUMBER="${GHOST_SMTP_PORT_NUMBER:-"${SMTP_PORT:-}"}"
GHOST_SMTP_PORT_NUMBER="${GHOST_SMTP_PORT_NUMBER:-"${GHOST_SMTP_PORT:-}"}"
export GHOST_SMTP_PORT_NUMBER="${GHOST_SMTP_PORT_NUMBER:-}" # only used during the first initialization
GHOST_SMTP_USER="${GHOST_SMTP_USER:-"${SMTP_USER:-}"}"
export GHOST_SMTP_USER="${GHOST_SMTP_USER:-}" # only used during the first initialization
GHOST_SMTP_PASSWORD="${GHOST_SMTP_PASSWORD:-"${SMTP_PASSWORD:-}"}"
export GHOST_SMTP_PASSWORD="${GHOST_SMTP_PASSWORD:-}" # only used during the first initialization
GHOST_SMTP_PROTOCOL="${GHOST_SMTP_PROTOCOL:-"${SMTP_PROTOCOL:-}"}"
export GHOST_SMTP_PROTOCOL="${GHOST_SMTP_PROTOCOL:-}" # only used during the first initialization

# Database configuration
export GHOST_DEFAULT_DATABASE_HOST="mysql" # only used at build time
GHOST_DATABASE_HOST="${GHOST_DATABASE_HOST:-"${MYSQL_HOST:-}"}"
export GHOST_DATABASE_HOST="${GHOST_DATABASE_HOST:-$GHOST_DEFAULT_DATABASE_HOST}" # only used during the first initialization
GHOST_DATABASE_PORT_NUMBER="${GHOST_DATABASE_PORT_NUMBER:-"${MYSQL_PORT_NUMBER:-}"}"
export GHOST_DATABASE_PORT_NUMBER="${GHOST_DATABASE_PORT_NUMBER:-3306}" # only used during the first initialization
GHOST_DATABASE_NAME="${GHOST_DATABASE_NAME:-"${MYSQL_DATABASE_NAME:-}"}"
export GHOST_DATABASE_NAME="${GHOST_DATABASE_NAME:-bitnami_ghost}" # only used during the first initialization
GHOST_DATABASE_USER="${GHOST_DATABASE_USER:-"${MYSQL_DATABASE_USER:-}"}"
export GHOST_DATABASE_USER="${GHOST_DATABASE_USER:-bn_ghost}" # only used during the first initialization
GHOST_DATABASE_PASSWORD="${GHOST_DATABASE_PASSWORD:-"${MYSQL_DATABASE_PASSWORD:-}"}"
export GHOST_DATABASE_PASSWORD="${GHOST_DATABASE_PASSWORD:-}" # only used during the first initialization
export GHOST_DATABASE_ENABLE_SSL="${GHOST_DATABASE_ENABLE_SSL:-no}" # only used during the first initialization
export GHOST_DATABASE_SSL_CA_FILE="${GHOST_DATABASE_SSL_CA_FILE:-}" # only used during the first initialization

# Custom environment variables may be defined below

I have no name!@ghost-blog-7cdf7f8967-ggbcb:/opt/bitnami/scripts$ ls -al /bitnami/ghost
total 28
drwxrwsr-x  4 1001 1001  4096 Feb 12 14:52 .
drwxr-xr-x  3 root root    19 Feb  9 20:15 ..
-rw-rw-r--  1 1001 1001   239 Feb 12 14:52 config.production.json
drwxrwsrwx 11 1001 1001  4096 Feb 12 14:52 content
drwxrws---  2 1001 1001 16384 Feb 12 14:50 lost+found

I have no name!@ghost-blog-7cdf7f8967-ggbcb:/opt/bitnami/scripts$ cd ghost
I have no name!@ghost-blog-7cdf7f8967-ggbcb:/opt/bitnami/scripts/ghost$ ./entrypoint.sh ./run.sh
ghost 15:01:15.54 INFO  ==>
ghost 15:01:15.54 INFO  ==> Welcome to the Bitnami ghost container
ghost 15:01:15.54 INFO  ==> Subscribe to project updates by watching https://github.com/bitnami/containers
ghost 15:01:15.54 INFO  ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
ghost 15:01:15.54 INFO  ==>
ghost 15:01:15.54 INFO  ==> Configuring libnss_wrapper

ghost 15:01:15.58 INFO  ==> ** Starting Ghost **

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

The `ghost run` command is used by the configured Ghost process manager and for debugging. If you're not running this to debug something, you should run `ghost start` instead.
[2024-02-12 15:01:20] INFO Ghost is running in production...
[2024-02-12 15:01:20] INFO Your site is now available on http://localhost:2368/
[2024-02-12 15:01:20] INFO Ctrl+C to shut down
[2024-02-12 15:01:20] INFO Ghost server started in 2.122s
node:events:495
      throw er; // Unhandled 'error' event
      ^

Error: EACCES: permission denied, open '/opt/bitnami/ghost/versions/5.79.1/content/logs/http___localhost_2368_production.error.log'
Emitted 'error' event on WriteStream instance at:
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -13,
  code: 'EACCES',
  syscall: 'open',
  path: '/opt/bitnami/ghost/versions/5.79.1/content/logs/http___localhost_2368_production.error.log'
}

Node.js v18.19.0

Thanks @druheendas for all the details. It's very useful for us and other users.

It seems the database does not match with the content stored by the Ghost pod (the database says Ghost has the file http___localhost_2368_production.error.log but it doesn't exist). Have you taken a look to this comment? Do you have any idea about the connection errors you faced in the first attempt?

Initial connection error was coming because DB was taking some time to come up. Right now am experimenting with external DB. same issue. I drop the schema create the schema again, then delete helm release, delete namespace, delete the PV. Then deploy again. But same Error.

I am sorry, I cannot reproduce the issue in a clean environment from zero. I can force the situation removing the persistent volumes, which is not a desired option. The steps in this comment describes how can we clean the environment.

I am worried about how did you reach this situation initially, have you upgraded your previous installation or are you facing this issue from the very begining?

bitnami / charts