Open toughcoding opened 1 year ago
There was related discussion for it on forum. https://discuss.elastic.co/t/keystore-password-file/339627/4
Pinging @elastic/es-distributed (Team:Distributed)
Pinging @elastic/es-delivery (Team:Delivery)
@elastic/es-security I'm having issues even trying to reproduce this. If I create a password-protected keystore, then bind mount it into the container and attempt to start ES it blows up during security auto-configuration. Is this a known issue?
Enter password for the elasticsearch keystore : Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:874)
at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
at java.base/java.nio.file.Files.move(Files.java:1429)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:514)
at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428)
at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:582)
at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:165)
at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:86)
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
at org.elasticsearch.cli.Command.main(Command.java:50)
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
Suppressed: java.lang.UnsupportedOperationException: Unsupported copy option
at java.base/sun.nio.fs.UnixFileSystem$Flags.fromMoveOptions(UnixFileSystem.java:500)
at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:861)
at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309)
at java.base/java.nio.file.Files.move(Files.java:1429)
at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:587)
I think this is related to https://github.com/elastic/elasticsearch/pull/76124
The issue is not with docker. It's a bug for adding string value to a password protected ES keystore. Assuming the keystore password is 123456, it can be reproduced with
COMMANDS="$(printf "%s\n%s" "123456" "abcdefg")"
echo "$COMMANDS" | ./bin/elasticsearch-keystore add -x 'foo'
# The following command shows empty for `foo`
./bin/elasticsearch-keystore show foo
Basically the terminal reader for keystore password eagerly reads everything and leaves nothing for the actual content. I think it needs attention from @elastic/es-core-infra Also ping @albertzaharovits since he was the author of the linked PR.
@elastic/es-security I'm having issues even trying to reproduce this. If I create a password-protected keystore, then bind mount it into the container and attempt to start ES it blows up during security auto-configuration. Is this a known issue?
Enter password for the elasticsearch keystore : Exception in thread "main" java.nio.file.FileSystemException: /usr/share/elasticsearch/config/elasticsearch.keystore.tmp -> /usr/share/elasticsearch/config/elasticsearch.keystore: Device or resource busy at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:874) at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309) at java.base/java.nio.file.Files.move(Files.java:1429) at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:514) at org.elasticsearch.common.settings.KeyStoreWrapper.save(KeyStoreWrapper.java:428) at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:582) at org.elasticsearch.server.cli.ServerCli.autoConfigureSecurity(ServerCli.java:165) at org.elasticsearch.server.cli.ServerCli.execute(ServerCli.java:86) at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54) at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) at org.elasticsearch.cli.Command.main(Command.java:50) at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64) Suppressed: java.lang.UnsupportedOperationException: Unsupported copy option at java.base/sun.nio.fs.UnixFileSystem$Flags.fromMoveOptions(UnixFileSystem.java:500) at java.base/sun.nio.fs.UnixFileSystem.move(UnixFileSystem.java:861) at java.base/sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:309) at java.base/java.nio.file.Files.move(Files.java:1429) at org.elasticsearch.xpack.security.cli.AutoConfigureNode.execute(AutoConfigureNode.java:587)
To make it work like you wish I am running first container with mounted volume into config and generate password protected keystore. Then secondly I am stopping that container and starting new container with mounted config catalog not just keystore file. So I am using volumes therefore not bind mounts. Elastic team do not recommend to bind mount keystore file mentioning it can cause issues.
I think this is related to #76124
The issue is not with docker. It's a bug for adding string value to a password protected ES keystore. Assuming the keystore password is 123456, it can be reproduced with
COMMANDS="$(printf "%s\n%s" "123456" "abcdefg")" echo "$COMMANDS" | ./bin/elasticsearch-keystore add -x 'foo' # The following command shows empty for `foo` ./bin/elasticsearch-keystore show foo
Basically the terminal reader for keystore password eagerly reads everything and leaves nothing for the actual content. I think it needs attention from @elastic/es-core-infra Also ping @albertzaharovits since he was the author of the linked PR.
Yes it is not due to docker itself but on high level running docker does not work as you might expect from documentation.
I was positive we had keystore tests for this exact scenario. However, upon closer inspection I found that the tests are bogus for a variety of reasons. Unfortunately there are a couple places where we use buffered readers which break this. Fixing it is non trivial; the internal Terminal abstraction needs more work to cleanup how we handle raw input.
Pinging @elastic/es-core-infra (Team:Core/Infra)
Its been a little while but I think I'm seeing this same issue when trying to use a precreated keystore, we're providing the password to the keystore in the KEYSTORE_PASSWORD
environment variable, but the entrypoint continues to prompt for a password.
If I exec into a running container I can see the variable is set, and if I then run /bin/elasticsearch
and enter the keystore password things proceed. But it's not respecting the environment variable as far as I can see. Is there a workaround in the mean time?
@da1910 You can check my article about this. Maybe you will find some inspiration from commands. Otherwise let's wait for Elastic to come back with suggestions.
Elasticsearch Version
8.9.0
Installed Plugins
No response
Java Version
bundled
OS Version
ubuntu:20.04 Linux 753c3b2604ff 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux
Problem Description
When starting Elasticsearch as a docker container with KEYSTORE_PASSWORD and ELASTIC_PASSWORD set and keystore being password secured, it does not working properly as bootstrap.password is not setup correctly. When checking value of bootstrap.password it is returning empty string. Thus command line 54
(echo "$COMMANDS" | elasticsearch-keystore add -x 'bootstrap.password')
is not causing bootstrap.password entry being created although running these commands manually in the container are successfull.
In Contrary Running with non-encrypted keystore making bootstrap.password to be setup properly like below
Steps to Reproduce
Create container only to make encrypted keystore
Run command to setup password for already created keystore setting password as 123456
docker exec -it elk elasticsearch-keystore passwd
Stop container
docker stop elk
Start another container with password variables
Checking if bootstrap password entry got created
Run command to display bootstrap.password that normally should be equal to ELASTIC_PASSWORD
docker exec -it elk elasticsearch-keystore show bootstrap.password
BUT it is returning empty string thus bug.
Logs (if relevant)
Beginning of logs after starting second container. Script asking twice for keystore password. docker run --rm \ --name elk \ -v esconf:/usr/share/elasticsearch/config \ -e ELASTIC_PASSWORD="123456" \ -e KEYSTORE_PASSWORD="123456" \ docker.elastic.co/elasticsearch/elasticsearch:8.9.0 Enter password for the elasticsearch keystore : Enter password for the elasticsearch keystore : {"@timestamp":"2023-08-01T19:08:33.302Z", "log.level": "INFO",