search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.15k stars 24.84k forks source link

bootstrap.StartupException NPE using `xpack.watcher.encrypt_sensitive_data` #30441

Closed jpcarey closed 1 year ago

jpcarey commented 6 years ago

Elasticsearch version: 6.2.4

Plugins installed: x-pack

Description of the problem including expected versus actual behavior: org.elasticsearch.bootstrap.StartupException NPE when configuring to use system_key.

Steps to reproduce:

  1. Fresh 6.2.4 + x-pack
  2. $ echo -e "xpack.watcher.encrypt_sensitive_data: true\n$(cat config/elasticsearch.yml)" > config/elasticsearch.yml
    1. $ bin/elasticsearch

Provide logs:

bin/elasticsearch
[2018-05-07T15:40:33,026][INFO ][o.e.n.Node               ] [] initializing ...
[2018-05-07T15:40:33,132][INFO ][o.e.e.NodeEnvironment    ] [7PzcpF8] using [1] data paths, mounts [[/ (/dev/disk1s1)]], net usable_space [146.4gb], net total_space [465.7gb], types [apfs]
[2018-05-07T15:40:33,133][INFO ][o.e.e.NodeEnvironment    ] [7PzcpF8] heap size [990.7mb], compressed ordinary object pointers [true]
[2018-05-07T15:40:33,204][INFO ][o.e.n.Node               ] node name [7PzcpF8] derived from node ID [7PzcpF8bRBySHM2VOtOGxA]; set [node.name] to override
[2018-05-07T15:40:33,204][INFO ][o.e.n.Node               ] version[6.2.4], pid[65893], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Mac OS X/10.13.4/x86_64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_65/25.65-b01]
[2018-05-07T15:40:33,204][INFO ][o.e.n.Node               ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/var/folders/v4/kjd2tdmj2fq27dbtp68114x40000gn/T/elasticsearch.qLK45OLc, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/Users/jared/builds/elasticsearch/elasticsearch-6.2.4, -Des.path.conf=/Users/jared/builds/elasticsearch/elasticsearch-6.2.4/config]
[2018-05-07T15:40:36,209][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [aggs-matrix-stats]
[2018-05-07T15:40:36,209][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [analysis-common]
[2018-05-07T15:40:36,209][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [ingest-common]
[2018-05-07T15:40:36,211][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [lang-expression]
[2018-05-07T15:40:36,211][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [lang-mustache]
[2018-05-07T15:40:36,212][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [lang-painless]
[2018-05-07T15:40:36,213][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [mapper-extras]
[2018-05-07T15:40:36,213][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [parent-join]
[2018-05-07T15:40:36,213][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [percolator]
[2018-05-07T15:40:36,214][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [rank-eval]
[2018-05-07T15:40:36,215][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [reindex]
[2018-05-07T15:40:36,215][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [repository-url]
[2018-05-07T15:40:36,215][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [transport-netty4]
[2018-05-07T15:40:36,216][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded module [tribe]
[2018-05-07T15:40:36,218][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-core]
[2018-05-07T15:40:36,218][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-deprecation]
[2018-05-07T15:40:36,218][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-graph]
[2018-05-07T15:40:36,219][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-logstash]
[2018-05-07T15:40:36,219][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-ml]
[2018-05-07T15:40:36,219][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-monitoring]
[2018-05-07T15:40:36,219][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-security]
[2018-05-07T15:40:36,219][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-upgrade]
[2018-05-07T15:40:36,219][INFO ][o.e.p.PluginsService     ] [7PzcpF8] loaded plugin [x-pack-watcher]
[2018-05-07T15:40:42,140][INFO ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/65912] [Main.cc@128] controller (64 bit): Version 6.2.4 (Build 524e7fe231abc1) Copyright (c) 2018 Elasticsearch BV
[2018-05-07T15:40:42,230][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.NullPointerException
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.4.jar:6.2.4]
Caused by: java.lang.NullPointerException
    at org.elasticsearch.common.io.Streams.readFully(Streams.java:214) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.common.io.Streams.readFully(Streams.java:208) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.xpack.core.watcher.crypto.CryptoService.readSystemKey(CryptoService.java:105) ~[?:?]
    at org.elasticsearch.xpack.core.watcher.crypto.CryptoService.<init>(CryptoService.java:93) ~[?:?]
    at org.elasticsearch.xpack.watcher.Watcher.createComponents(Watcher.java:286) ~[?:?]
    at org.elasticsearch.node.Node.lambda$new$7(Node.java:397) ~[elasticsearch-6.2.4.jar:6.2.4]
    at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) ~[?:1.8.0_65]
    at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_65]
    at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_65]
    at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_65]
    at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_65]
    at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_65]
    at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_65]
    at org.elasticsearch.node.Node.<init>(Node.java:400) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.node.Node.<init>(Node.java:246) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.4.jar:6.2.4]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.4.jar:6.2.4]
    ... 6 more
elasticmachine commented 6 years ago

Pinging @elastic/es-core-infra

spinscale commented 6 years ago

Just to make sure I properly understand: Did you leave out bin/elasticsearch-keystore add-file xpack.watcher.encryption_key <filepath>/system_key on purpose or is the exception thrown the bug you reported, when you did not call this?

jpcarey commented 6 years ago

@spinscale intentionally skipped adding the system_key to the keystore. I would expect an error that it wasn’t properly configured, rather than a NPE. On Tue, May 8, 2018 at 2:28 AM Alexander Reelsen notifications@github.com wrote:

Just to make sure I properly understand: Did you leave out bin/elasticsearch-keystore add-file xpack.watcher.encryption_key /system_key on purpose or is the exception thrown the bug you reported, when you did not call this?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/elastic/elasticsearch/issues/30441#issuecomment-387325860, or mute the thread https://github.com/notifications/unsubscribe-auth/AE4aBpR9NdFI-l3KhIklqdCvIaptZqXsks5twVc4gaJpZM4T1tAZ .

willemdh commented 5 years ago

Just had this same error.. After adding the system key in the elasticsearch keystore the issue was solved.

ayushmathur86 commented 5 years ago

@willemdh how did you add the key ? When I tried it, the key and encryption were not taken into account unless I don't restart the node. However, on restarting the nodes, I still get the same NPE stack since the key is lost, i.e. it's not persistent.

willemdh commented 5 years ago

Copied the system key from another node and imported it