search

while1eq1 / elasticsearch-kubernetes-searchguard

ElasticSearch 6.4+ based docker image with SearchGuard 6 for TLS Transport/REST for use in Kubernetes
3 stars 3 forks source link

Elasticsearch client container doesn't start up #2

Open jacobreid opened 5 years ago

jacobreid commented 5 years ago

When running with elasticsearch-operator 0.3.0, the es-client containers never reach a ready state as the searchguard plugin fails to initialise as it is looking for the cert at the wrong path.

$ kubectl logs es-client-logs-dev-5bb5f54f8b-6d2bz 
Starting ElasticSearch 6.4.1
Changing ownership of /elasticsearch folder
chown: ./config/certs/..2019_03_12_16_20_28.710076595/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/truststore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/node.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/kibana-key.pem: Read-only file system
Changing ownership of /data folder
chown: ./config/certs/..2019_03_12_16_20_28.710076595/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/truststore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/node.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_20_28.710076595/kibana-key.pem: Read-only file system
Waiting for Elasticsearch to become ready before running sgadmin...
[2019-03-12T16:26:33,879][INFO ][o.e.n.Node               ] [da1352b9-f984-4b35-bf0a-3397991e0ea4] initializing ...
[2019-03-12T16:26:33,949][INFO ][o.e.e.NodeEnvironment    ] [da1352b9-f984-4b35-bf0a-3397991e0ea4] using [1] data paths, mounts [[/data (/dev/nvme0n1p2)]], net usable_space [89gb], net total_space [119.9gb], types [ext4]
[2019-03-12T16:26:33,950][INFO ][o.e.e.NodeEnvironment    ] [da1352b9-f984-4b35-bf0a-3397991e0ea4] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-03-12T16:26:33,951][INFO ][o.e.n.Node               ] [da1352b9-f984-4b35-bf0a-3397991e0ea4] node name [da1352b9-f984-4b35-bf0a-3397991e0ea4], node ID [X0j85xDkTJSwUDIdWeYYSg]
[2019-03-12T16:26:33,951][INFO ][o.e.n.Node               ] [da1352b9-f984-4b35-bf0a-3397991e0ea4] version[6.4.1], pid[19], build[default/tar/e36acdb/2018-09-13T22:18:07.696808Z], OS[Linux/4.9.0-7-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-03-12T16:26:33,951][INFO ][o.e.n.Node               ] [da1352b9-f984-4b35-bf0a-3397991e0ea4] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms1024m, -Xmx1024m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-03-12T16:26:35,373][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin
[2019-03-12T16:26:35,526][INFO ][c.f.s.SearchGuardPlugin  ] ES Config path is /elasticsearch/config
[2019-03-12T16:26:35,566][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2019-03-12T16:26:35,571][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2019-03-12T16:26:35,636][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [da1352b9-f984-4b35-bf0a-3397991e0ea4] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: org.elasticsearch.ElasticsearchException: Unable to read /elasticsearch/config/certs/node-key.pkcs8.pem (/elasticsearch/config/certs/node-key.pkcs8.pem). Please make sure this files exists and is readable regarding to permissions. Property: searchguard.ssl.transport.pemkey_filepath
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.checkPath(DefaultSearchGuardKeyStore.java:809) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.resolve(DefaultSearchGuardKeyStore.java:210) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:327) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
jacobreid commented 5 years ago

If I replace the cert's path in the config file, searchguard still fails to start up as expected:

$  git diff config/elasticsearch.yml
diff --git a/config/elasticsearch.yml b/config/elasticsearch.yml
index e99ea89..98c2062 100644
--- a/config/elasticsearch.yml
+++ b/config/elasticsearch.yml
@@ -44,13 +44,13 @@ searchguard:
     - "CN=node,OU=elasticsearch cluster,O=autogenerated,L=operator"
   ssl:
     transport:
-      pemkey_filepath: certs/node-key.pkcs8.pem
+      pemkey_filepath: certs/node-key.pem
       pemcert_filepath: certs/node.pem
       pemtrustedcas_filepath: certs/ca.pem
       enforce_hostname_verification: false
       enabled: ${SEARCHGUARD_SSL_TRANSPORT_ENABLED}
     http:
-      pemkey_filepath: certs/node-key.pkcs8.pem
+      pemkey_filepath: certs/node-key.pem
       pemcert_filepath: certs/node.pem
       pemtrustedcas_filepath: certs/ca.pem
       enabled: ${SEARCHGUARD_SSL_HTTP_ENABLED}
$ kubectl logs es-client-logs-dev-7cc565cc5f-fps6k
Starting ElasticSearch 6.4.1
Changing ownership of /elasticsearch folder
chown: ./config/certs/..2019_03_12_16_34_44.221669483/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/node.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/truststore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/kibana-key.pem: Read-only file system
Changing ownership of /data folder
chown: ./config/certs/..2019_03_12_16_34_44.221669483/ca-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/node-keystore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/kibana.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/cerebro-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/ca.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/node.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/node-key.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/cerebro.pem: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/truststore.jks: Read-only file system
chown: ./config/certs/..2019_03_12_16_34_44.221669483/kibana-key.pem: Read-only file system
Waiting for Elasticsearch to become ready before running sgadmin...
[2019-03-12T16:34:50,701][INFO ][o.e.n.Node               ] [9231192c-8192-429f-89db-907092fe30b6] initializing ...
[2019-03-12T16:34:50,751][INFO ][o.e.e.NodeEnvironment    ] [9231192c-8192-429f-89db-907092fe30b6] using [1] data paths, mounts [[/data (/dev/nvme0n1p2)]], net usable_space [78.6gb], net total_space [119.9gb], types [ext4]
[2019-03-12T16:34:50,751][INFO ][o.e.e.NodeEnvironment    ] [9231192c-8192-429f-89db-907092fe30b6] heap size [1007.3mb], compressed ordinary object pointers [true]
[2019-03-12T16:34:50,752][INFO ][o.e.n.Node               ] [9231192c-8192-429f-89db-907092fe30b6] node name [9231192c-8192-429f-89db-907092fe30b6], node ID [qRwMLZhqSQWTh1lCok-Atw]
[2019-03-12T16:34:50,752][INFO ][o.e.n.Node               ] [9231192c-8192-429f-89db-907092fe30b6] version[6.4.1], pid[19], build[default/tar/e36acdb/2018-09-13T22:18:07.696808Z], OS[Linux/4.9.0-7-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12]
[2019-03-12T16:34:50,752][INFO ][o.e.n.Node               ] [9231192c-8192-429f-89db-907092fe30b6] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Xms1024m, -Xmx1024m, -Des.path.home=/elasticsearch, -Des.path.conf=/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
[2019-03-12T16:34:52,267][INFO ][o.e.p.p.PrometheusExporterPlugin] starting Prometheus exporter plugin
[2019-03-12T16:34:52,439][INFO ][c.f.s.SearchGuardPlugin  ] ES Config path is /elasticsearch/config
[2019-03-12T16:34:52,480][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2019-03-12T16:34:52,486][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2019-03-12T16:34:52,552][ERROR][c.f.s.s.DefaultSearchGuardKeyStore] Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.
[2019-03-12T16:34:52,623][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [9231192c-8192-429f-89db-907092fe30b6] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:140) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
    at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86) ~[elasticsearch-6.4.1.jar:6.4.1]
Caused by: java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:607) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: org.elasticsearch.ElasticsearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /elasticsearch/config/certs/node-key.pem
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:346) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /elasticsearch/config/certs/node-key.pem
    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:726) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:333) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045) ~[?:?]
    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:726) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:333) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169) ~[?:?]
    at java.security.KeyFactory.generatePrivate(KeyFactory.java:372) ~[?:1.8.0_191]
    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043) ~[?:?]
    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:726) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:333) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352) ~[?:?]
    at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357) ~[?:?]
    at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73) ~[?:?]
    at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237) ~[?:?]
    at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165) ~[?:?]
    at java.security.KeyFactory.generatePrivate(KeyFactory.java:372) ~[?:1.8.0_191]
    at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043) ~[?:?]
    at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265) ~[?:?]
    at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:726) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:333) ~[?:?]
    at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:151) ~[?:?]
    at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193) ~[?:?]
    at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_191]
    at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:598) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundle(PluginsService.java:549) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:464) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:156) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:315) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.node.Node.<init>(Node.java:256) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:213) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:326) ~[elasticsearch-6.4.1.jar:6.4.1]
    at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-6.4.1.jar:6.4.1]
    ... 6 more
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
Waiting for Elasticsearch to become ready before running sgadmin...
jacobreid commented 5 years ago

Logs from elasticsearch-operator show it is attempting to convert the node cert to pkcs12 as found in https://github.com/upmc-enterprises/elasticsearch-operator/blob/master/pkg/k8sutil/certs.go#L213 but the resulting cert is not found in the certs directory on either the es-client or es-data pods:

bash-4.4# ls /elasticsearch/config/certs/
ca-key.pem         cerebro-key.pem    kibana-key.pem     node-key.pem       node.pem
ca.pem             cerebro.pem        kibana.pem         node-keystore.jks  truststore.jks
time="2019-03-14T15:19:58Z" level=info msg="Process Elasticsearch Event ADDED"
time="2019-03-14T15:19:58Z" level=info msg="--------> Received ElasticSearch Event!"
time="2019-03-14T15:19:58Z" level=info msg="-----> Stop scheduler logs-dev-development-sre"
time="2019-03-14T15:19:58Z" level=info msg="Found cluster: logs-dev"
time="2019-03-14T15:19:58Z" level=info msg="use-ssl true"
time="2019-03-14T15:19:58Z" level=info msg="Using [docker-registry.********.com/production/elasticsearch-kubernetes-searchguard:master] as image for es cluster"
time="2019-03-14T15:19:58Z" level=info msg="use-ssl true"
time="2019-03-14T15:19:58Z" level=info msg="Creating new certs!"
time="2019-03-14T15:19:58Z" level=info msg="Creating ca cert..."
time="2019-03-14T15:20:03Z" level=info msg="Creating node cert..."
time="2019-03-14T15:20:07Z" level=info msg="Creating kibana cert..."
time="2019-03-14T15:20:11Z" level=info msg="Creating cerebro cert..."
time="2019-03-14T15:20:18Z" level=info msg="Converting node to pkcs12..."
time="2019-03-14T15:20:18Z" level=info msg="Converting ca cert to jks..."
time="2019-03-14T15:20:20Z" level=info msg="Converting node cert to jks..."
time="2019-03-14T15:20:26Z" level=info msg="Discovery Service not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="Service es-data-svc-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="elasticsearch-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="cerebro-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="kibana-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="Deployment es-client-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="StatefulSet es-master-logs-dev-us-east-1b not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="StatefulSet es-master-logs-dev-us-east-1c not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="StatefulSet es-master-logs-dev-us-east-1d not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="StatefulSet es-data-logs-dev-us-east-1b not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="StatefulSet es-data-logs-dev-us-east-1c not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="StatefulSet es-data-logs-dev-us-east-1d not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="kibana-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="Deployment cerebro-logs-dev not found, creating..."
time="2019-03-14T15:20:26Z" level=info msg="--------> ElasticSearch Event finished!"
time="2019-03-14T15:27:13Z" level=info msg="Process Elasticsearch Event MODIFIED"
time="2019-03-14T15:27:13Z" level=info msg="--------> Received ElasticSearch Event!"
time="2019-03-14T15:27:13Z" level=info msg="-----> Stop scheduler logs-dev-development-sre"
time="2019-03-14T15:27:13Z" level=info msg="Found cluster: logs-dev"
time="2019-03-14T15:27:13Z" level=info msg="use-ssl true"
time="2019-03-14T15:27:13Z" level=info msg="Using [docker-registry.********.com/production/elasticsearch-kubernetes-searchguard:master] as image for es cluster"
time="2019-03-14T15:27:13Z" level=info msg="use-ssl true"
time="2019-03-14T15:27:14Z" level=info msg="--------> ElasticSearch Event finished!"
imriss commented 5 years ago

@jacobreid Have you found a solution? Thanks