Enabling readOnlyFileSystem with a multi-node elasticsearch cluster fails on startup

amitonlentra commented 8 hours ago

Name and Version

bitnami/elasticsearch: 19.13.15

What architecture are you using?

amd64

What steps will reproduce the bug?

We are deploying elasticsearch v8.9.2 through zeebe (open source workflow engine from camunda team). The helm chart used is version: 19.13.15 by camunda.

To tighten security our requirement is to run elasticsearch with minimal privileges and on read only file system. We configured it with the below helm config but the deployment fails with an startup error "Exception in thread "main" java.nio.file.FileSystemException: /opt/bitnami/elasticsearch/config/elasticsearch.keystore.tmp: Read-only file system". The complete stacktrace is shared below.

We looked into a few previously reported issues but could not find a solution. Kindly suggest a solution that works.

Are you using any custom parameters or values?

Below is the helm config to tighten security.

elasticsearch:
  enabled: true
  metrics:
    enabled: true
    resources:
      requests:
        cpu: 500m
        memory: 512Mi
      limits:
        cpu: 1
        memory: 512Mi
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
  master:
    masterOnly: false
    replicaCount: 2
    containerSecurityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
  image:
    repository: "bitnami/elasticsearch"
    tag: "8.9.2"
 resources:
    requests:
      cpu: 1.5
      memory: 2Gi
    limits:
      cpu: 1.5
      memory: 2Gi
  volumeClaimTemplate:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "local-path"
    resources:
      requests:
        storage: "20Gi"

What is the expected behavior?

Elasticsearch should be deployed and executed without errors

What do you see instead?

Complete stacktrace

Defaulted container "elasticsearch" out of: elasticsearch, sysctl (init)
elasticsearch 12:35:18.05 
elasticsearch 12:35:18.06 Welcome to the Bitnami elasticsearch container
elasticsearch 12:35:18.06 Subscribe to project updates by watching https://github.com/bitnami/containers
elasticsearch 12:35:18.07 Submit issues and feature requests at https://github.com/bitnami/containers/issues
elasticsearch 12:35:18.07 
elasticsearch 12:35:18.07 INFO  ==> ** Starting Elasticsearch setup **
elasticsearch 12:35:18.17 WARN  ==> Setting ELASTICSEARCH_IS_DEDICATED_NODE is disabled.
elasticsearch 12:35:18.18 WARN  ==> ELASTICSEARCH_NODE_ROLES will be ignored and Elasticsearch will asume all different roles.
elasticsearch 12:35:18.19 INFO  ==> Configuring/Initializing Elasticsearch...
elasticsearch 12:35:18.27 INFO  ==> Custom configuration file detected, using it...
elasticsearch 12:35:18.27 WARN  ==> The JVM options configuration files are not writable. Configurations based on environment variables will not be applied
elasticsearch 12:35:18.36 INFO  ==> ** Elasticsearch setup finished! **

elasticsearch 12:35:18.41 INFO  ==> ** Starting Elasticsearch **
warning: ignoring JAVA_HOME=/opt/bitnami/java; using ES_JAVA_HOME
Exception in thread "main" java.nio.file.FileSystemException: /opt/bitnami/elasticsearch/config/elasticsearch.keystore.tmp: Read-only file system
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
    at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484)
    at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:394)
    at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:387)
    at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)
    at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:438)
    at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
    at org.elasticsearch.common.settings.KeyStoreWrapper.bootstrap(KeyStoreWrapper.java:233)
    at org.elasticsearch.server.cli.KeyStoreLoader.bootstrap(KeyStoreLoader.java:37)
    at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:88)
    at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
    at org.elasticsearch.cli.Command.main(Command.java:50)
    at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

carrodher commented 7 hours ago

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

If you think that's not the case and want to contribute a solution, we'd like to invite you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Please feel free to contact us if you have any questions or need assistance.

Suppose you have any questions about the application, customizing its content, or technology and infrastructure usage. In that case, we highly recommend that you refer to the forums and user guides provided by the project responsible for the application or technology.

With that said, we'll keep this ticket open until the stale bot automatically closes it, in case someone from the community contributes valuable insights.

amitonlentra commented 6 hours ago

Thanks @carrodher for the reply. I have a follow up question

Hi, the issue may not be directly related to the Bitnami container image/Helm chart, but rather to how the application is being utilized, configured in your specific environment, or tied to a particular scenario that is not easy to reproduce on our side.

I am not sure if I follow you completely.

Since these attributes readOnlyRootFilesystem, allowPrivilegeEscalation are defined by the bitnami helm chart, I am assuming they should be tested and supported by the bitnami helm chart team. Please correct me if my understanding is incorrect.
There is no specific environment or scenario needed to reproduce this. If you just use the above config, you would be able to see the error.

bitnami / charts