elasticsearch.keystore: Device or resource busy

[SashaShcherbyna]

Chart version: 8.5.1

Kubernetes version: 1.23.12-gke.1600

Kubernetes provider: GKE (Google Kubernetes Engine)

Helm Version: version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.19"}

Describe the bug: Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy

Steps to reproduce:

1. Create the secret

   kubectl create secret generic elk-backup --from-file=gcs.client.elk-backup.credentials_file=./elk-backup.json

2. To add these secrets to the keystore:

   keystore:
   - secretName: elk-backup 

   Expected behavior: mount the secret in keystore should work

Provide logs and/or server output (if relevant):

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:420)
at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266)
at java.base/java.nio.file.Files.move(Files.java:1430)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:498)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:412)
at org.elasticsearch.cli.keystore.AddStringKeyStoreCommand.executeCommand(AddStringKeyStoreCommand.java:102)
at org.elasticsearch.cli.keystore.BaseKeyStoreCommand.execute(BaseKeyStoreCommand.java:64)
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
at org.elasticsearch.cli.Command.main(Command.java:50)
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

Any additional context: I can seeing this error in pod with role master

[TanguyPatte]

I'm not sure but maybe it's because the keystore file is mounted as a subpath in the chart

- name: keystore
  mountPath: /usr/share/elasticsearch/config/elasticsearch.keystore
  subPath: elasticsearch.keystore

I don't know how elasticsearch-keystore tool works but it looks like it's trying to replace elasticsearch.keystore file by elasticsearch.keystore.tmp file instead of replacing the content of it.

I reproduce the issue by trying to do the following command:

mv elasticsearch.keystore.tmp elasticsearch.keystore
mv: cannot move 'elasticsearch.keystore.tmp' to 'elasticsearch.keystore': Device or resource busy

cp command works just fine

cp elasticsearch.keystore.tmp elasticsearch.keystore

I see 2 possibilities to fix that:

• change the way elasticsearch-keystore write content in keystore file
• add the possibility to specify a custom path for the keystore file and mount it in is own directory to prevent the use of subpath

[adrifermo]

Hi @TanguyPatte I've tried changing cp -a to cp just, it works in that InitContainer "keystore", however the same issue not appears on the principal container, I'm passing for the same issue that Sasha mention on GKE Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:416) at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266) at java.base/java.nio.file.Files.move(Files.java:1432) at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:498) at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:412) at org.elasticsearch.cli.keystore.AddStringKeyStoreCommand.executeCommand(AddStringKeyStoreCommand.java:102) at org.elasticsearch.cli.keystore.BaseKeyStoreCommand.execute(BaseKeyStoreCommand.java:64) at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.Command.main(Command.java:50) at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

[Drookoo]

seems like there is something wrong with the current statefulset.yaml that causes this issue.

FWIW deploying with this template works well for elasticsearch 8.6.2 : https://github.com/elastic/helm-charts/blob/d4e9f6bc47cf7f7ad4dfaaec102e1327d8a345e3/elasticsearch/templates/statefulset.yaml

but there may be more recent iterations that may work.

[ppatcha]

Chart version: 8.5.1

Kubernetes version: 1.23.12-gke.1600

Kubernetes provider: GKE (Google Kubernetes Engine)

Helm Version: version.BuildInfo{Version:"v3.9.3", GitCommit:"414ff28d4029ae8c8b05d62aa06c7fe3dee2bc58", GitTreeState:"clean", GoVersion:"go1.19"}

Describe the bug: Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy

Steps to reproduce:

1. Create the secret

kubectl create secret generic elk-backup --from-file=gcs.client.elk-backup.credentials_file=./elk-backup.json

2. To add these secrets to the keystore:

keystore:
  - secretName: elk-backup 

Expected behavior: mount the secret in keystore should work

Provide logs and/or server output (if relevant):

Be careful to obfuscate every secrets (credentials, token, public IP, ...) that could be visible in the output before copy-pasting

Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:420)
at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:266)
at java.base/java.nio.file.Files.move(Files.java:1430)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:498)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:412)
at org.elasticsearch.cli.keystore.AddStringKeyStoreCommand.executeCommand(AddStringKeyStoreCommand.java:102)
at org.elasticsearch.cli.keystore.BaseKeyStoreCommand.execute(BaseKeyStoreCommand.java:64)
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
at org.elasticsearch.cli.Command.main(Command.java:50)
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)

Any additional context: I can seeing this error in pod with role master

@SashaShcherbyna I have got the same error, Did you have a solution to resolve this issue?

[ppatcha]

I can resolve this issue because I have not set ELASTIC_PASSWORD with my own credential. Just add

extraEnvs:
  - name: ELASTIC_PASSWORD
    valueFrom:
      secretKeyRef:
        name: elastic-config-credentials
        key: password

secret:
  enabled: false

and then add the keystore just normal.