search

opendistro-for-elasticsearch / sample-code

👋 Welcome to the Open Distro sample-code area. Share your great ideas and code samples with the Open Distro Community.
https://github.com/opendistro-for-elasticsearch/sample-code
Apache License 2.0
278 stars 78 forks source link

Zen discovery doesn't work #75

Closed ngtrunghuan closed 3 years ago

ngtrunghuan commented 5 years ago

Hi,

Big fan here. Been meaning to use ODFE since you guys released the first version, and now I just got the chance.

However, there's a very strange issue. I can't seem to use the docker-compose.yml here to produce a 2-node ODFE cluster on Docker running off a REHL server (2vCPU, 8GB RAM).

https://aws.amazon.com/blogs/opensource/running-open-distro-for-elasticsearch/

Expected: 2-node cluster ODFE What happened: 2 single-node clusters, with odfe-node2 cannot be accessed anywhere outside the container (due to unbound port)

Here is the _cat/nodes?v when using the dev tool on Kibana. ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 172.21.0.3 41 97 44 2.96 4.99 6.67 mdi * iyJUkY9

I can docker exec -it <containerId> /bin/bash into each container and ping the other host (for e.g. ping odfe-node1 in container for odfe-node2 and vice versa). However, the zen discovery won't even be Thanks a lot!

This it the log for odfe-node2

[root@sin0r3app2183xd odfe]# docker logs 1fc91edf4ee6
OpenDistro for Elasticsearch Security Demo Installer
 ** Warning: Do not use on production or public reachable systems **
Basedir: /usr/share/elasticsearch
Elasticsearch install type: rpm/deb on CentOS Linux release 7.6.1810 (Core)
Elasticsearch config dir: /usr/share/elasticsearch/config
Elasticsearch config file: /usr/share/elasticsearch/config/elasticsearch.yml
Elasticsearch bin dir: /usr/share/elasticsearch/bin
Elasticsearch plugins dir: /usr/share/elasticsearch/plugins
Elasticsearch lib dir: /usr/share/elasticsearch/lib
Detected Elasticsearch Version: x-content-6.7.1
Detected Open Distro Security Version: 0.9.0.0

### Success
### Execute this script now on all your nodes and then start all nodes
### Open Distro Security will be automatically initialized.
### If you like to change the runtime configuration
### change the files in ../securityconfig and execute:
"/usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh" -cd "/usr/share/elasticsearch/plugins/opendistro_security/securityconfig" -icl -key "/usr/share/elasticsearch/config/kirk-key.pem" -cert "/usr/share/elasticsearch/config/kirk.pem" -cacert "/usr/share/elasticsearch/config/root-ca.pem" -nhnv
### or run ./securityadmin_demo.sh
### To use the Security Plugin ConfigurationGUI
### To access your secured cluster open https://<hostname>:<HTTP port> and log in with admin/admin.
### (Ignore the SSL certificate warning because we installed self-signed demo certificates)
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2019-06-06T15:59:31,791][INFO ][o.e.e.NodeEnvironment    ] [WbbXwRt] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda2)]], net usable_space [50gb], net total_space [79.4gb], types [xfs]
[2019-06-06T15:59:31,797][INFO ][o.e.e.NodeEnvironment    ] [WbbXwRt] heap size [495.3mb], compressed ordinary object pointers [true]
[2019-06-06T15:59:31,801][INFO ][o.e.n.Node               ] [WbbXwRt] node name derived from node ID [WbbXwRtEQwGkmkAjzk7Enw]; set [node.name] to override
[2019-06-06T15:59:31,801][INFO ][o.e.n.Node               ] [WbbXwRt] version[6.7.1], pid[1], build[oss/tar/2f32220/2019-04-02T15:59:27.961366Z], OS[Linux/3.10.0-957.1.3.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
[2019-06-06T15:59:31,802][INFO ][o.e.n.Node               ] [WbbXwRt] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-1856553924822379077, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Djava.security.policy=file:///usr/share/elasticsearch/plugins/opendistro_performance_analyzer/pa_config/es_security.policy, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Des.cgroups.hierarchy.override=/, -Xms512m, -Xmx512m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar]
[2019-06-06T15:59:39,418][INFO ][c.a.o.e.p.c.PluginSettings] [WbbXwRt] loading config ...
[2019-06-06T15:59:39,419][INFO ][c.a.o.e.p.c.PluginSettings] [WbbXwRt] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1
[2019-06-06T15:59:42,252][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] ES Config path is /usr/share/elasticsearch/config
[2019-06-06T15:59:42,969][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2019-06-06T15:59:44,251][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] JVM supports TLSv1.3
[2019-06-06T15:59:44,252][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2019-06-06T15:59:47,408][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] TLS Transport Client Provider : JDK
[2019-06-06T15:59:47,408][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] TLS Transport Server Provider : JDK
[2019-06-06T15:59:47,408][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] TLS HTTP Provider             : JDK
[2019-06-06T15:59:47,408][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2, TLSv1.1]
[2019-06-06T15:59:47,409][INFO ][c.a.o.s.s.DefaultOpenDistroSecurityKeyStore] [WbbXwRt] Enabled TLS protocols for HTTP layer      : [TLSv1.3, TLSv1.2, TLSv1.1]
[2019-06-06T15:59:50,894][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] Clustername: odfe-cluster
[2019-06-06T15:59:51,089][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] Directory /usr/share/elasticsearch/config has insecure file permissions (should be 0700)
[2019-06-06T15:59:51,089][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/elasticsearch.yml has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,097][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/log4j2.properties has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,097][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/kirk.pem has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,097][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/esnode.pem has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,098][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/root-ca.pem has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,098][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/esnode-key.pem has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,098][WARN ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] File /usr/share/elasticsearch/config/kirk-key.pem has insecure file permissions (should be 0600)
[2019-06-06T15:59:51,854][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [aggs-matrix-stats]
[2019-06-06T15:59:51,855][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [analysis-common]
[2019-06-06T15:59:51,855][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [ingest-common]
[2019-06-06T15:59:51,855][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [ingest-geoip]
[2019-06-06T15:59:51,855][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [ingest-user-agent]
[2019-06-06T15:59:51,855][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [lang-expression]
[2019-06-06T15:59:51,855][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [lang-mustache]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [lang-painless]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [mapper-extras]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [parent-join]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [percolator]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [rank-eval]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [reindex]
[2019-06-06T15:59:51,856][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [repository-url]
[2019-06-06T15:59:51,857][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [transport-netty4]
[2019-06-06T15:59:51,874][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded module [tribe]
[2019-06-06T15:59:51,875][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded plugin [opendistro_alerting]
[2019-06-06T15:59:51,875][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded plugin [opendistro_performance_analyzer]
[2019-06-06T15:59:51,875][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded plugin [opendistro_security]
[2019-06-06T15:59:51,875][INFO ][o.e.p.PluginsService     ] [WbbXwRt] loaded plugin [opendistro_sql]
[2019-06-06T15:59:51,950][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml
[2019-06-06T16:00:16,787][INFO ][c.a.o.s.a.i.AuditLogImpl ] [WbbXwRt] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES]
[2019-06-06T16:00:16,788][INFO ][c.a.o.s.a.i.AuditLogImpl ] [WbbXwRt] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES]
[2019-06-06T16:00:16,788][INFO ][c.a.o.s.a.i.AuditLogImpl ] [WbbXwRt] Configured Users to ignore: [kibanaserver]
[2019-06-06T16:00:16,788][INFO ][c.a.o.s.a.i.AuditLogImpl ] [WbbXwRt] Configured Users to ignore for read compliance events: [kibanaserver]
[2019-06-06T16:00:16,788][INFO ][c.a.o.s.a.i.AuditLogImpl ] [WbbXwRt] Configured Users to ignore for write compliance events: [kibanaserver]
[2019-06-06T16:00:16,841][INFO ][c.a.o.s.a.i.AuditLogImpl ] [WbbXwRt] Message routing enabled: true
[2019-06-06T16:00:16,847][WARN ][c.a.o.s.c.ComplianceConfig] [WbbXwRt] If you plan to use field masking pls configure opendistro_security.compliance.salt to be a random string of 16 chars length identical on all nodes
[2019-06-06T16:00:16,847][INFO ][c.a.o.s.c.ComplianceConfig] [WbbXwRt] PII configuration [auditLogPattern=org.joda.time.format.DateTimeFormatter@65b73689,  auditLogIndex=null]: {}
[2019-06-06T16:00:18,323][INFO ][o.e.d.DiscoveryModule    ] [WbbXwRt] using discovery type [zen] and host providers [settings]
[2019-06-06T16:00:21,618][INFO ][c.a.o.e.p.h.c.PerformanceAnalyzerConfigAction] [WbbXwRt] PerformanceAnalyzer Enabled: true
Registering Handler
[2019-06-06T16:00:21,838][INFO ][o.e.n.Node               ] [WbbXwRt] initialized
[2019-06-06T16:00:21,846][INFO ][o.e.n.Node               ] [WbbXwRt] starting ...
[2019-06-06T16:00:22,722][INFO ][o.e.t.TransportService   ] [WbbXwRt] publish_address {172.21.0.2:9300}, bound_addresses {0.0.0.0:9300}
[2019-06-06T16:00:22,760][INFO ][o.e.b.BootstrapChecks    ] [WbbXwRt] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-06-06T16:00:22,815][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [WbbXwRt] Check if .opendistro_security index exists ...
[2019-06-06T16:00:23,476][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector MasterServiceMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:23,514][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NodeDetails is still in progress, so skipping this Interval
[2019-06-06T16:00:23,514][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NetworkInterfaceCollector is still in progress, so skipping this Interval
[2019-06-06T16:00:23,515][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector ThreadPoolMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:23,515][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector HeapMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:23,515][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector OSMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:23,515][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NodeStatsMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:23,515][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector MasterServiceEventMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:23,516][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector CircuitBreaker is still in progress, so skipping this Interval
[2019-06-06T16:00:23,516][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector DisksCollector is still in progress, so skipping this Interval
[2019-06-06T16:00:23,516][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NetworkE2ECollector is still in progress, so skipping this Interval
[2019-06-06T16:00:24,480][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector MasterServiceMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:24,480][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NodeDetails is still in progress, so skipping this Interval
[2019-06-06T16:00:24,480][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NetworkInterfaceCollector is still in progress, so skipping this Interval
[2019-06-06T16:00:24,480][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector ThreadPoolMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:24,480][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector HeapMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:24,481][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector OSMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:24,481][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NodeStatsMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:24,481][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector MasterServiceEventMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:24,481][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector CircuitBreaker is still in progress, so skipping this Interval
[2019-06-06T16:00:24,481][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector DisksCollector is still in progress, so skipping this Interval
[2019-06-06T16:00:24,481][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NetworkE2ECollector is still in progress, so skipping this Interval
[2019-06-06T16:00:25,482][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector OSMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:25,482][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector NetworkE2ECollector is still in progress, so skipping this Interval
[2019-06-06T16:00:26,461][INFO ][o.e.c.s.MasterService    ] [WbbXwRt] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {WbbXwRt}{WbbXwRtEQwGkmkAjzk7Enw}{iqod2-4-Rl2JEtmTTwhiMg}{172.21.0.2}{172.21.0.2:9300}
[2019-06-06T16:00:26,542][INFO ][o.e.c.s.ClusterApplierService] [WbbXwRt] new_master {WbbXwRt}{WbbXwRtEQwGkmkAjzk7Enw}{iqod2-4-Rl2JEtmTTwhiMg}{172.21.0.2}{172.21.0.2:9300}, reason: apply cluster state (from master [master {WbbXwRt}{WbbXwRtEQwGkmkAjzk7Enw}{iqod2-4-Rl2JEtmTTwhiMg}{172.21.0.2}{172.21.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics (file:/usr/share/elasticsearch/plugins/opendistro_performance_analyzer/opendistro_performance_analyzer-0.9.0.0.jar) to field java.util.concurrent.ThreadPoolExecutor.workers
WARNING: Please consider reporting this to the maintainers of com.amazon.opendistro.elasticsearch.performanceanalyzer.collectors.MasterServiceEventMetrics
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2019-06-06T16:00:27,061][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [WbbXwRt] .opendistro_security index does not exist yet, so we create a default config
[2019-06-06T16:00:27,136][INFO ][o.e.h.n.Netty4HttpServerTransport] [WbbXwRt] publish_address {172.21.0.2:9200}, bound_addresses {0.0.0.0:9200}
[2019-06-06T16:00:27,136][INFO ][o.e.n.Node               ] [WbbXwRt] started
[2019-06-06T16:00:27,136][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [WbbXwRt] 4 Open Distro Security modules loaded so far: [Module [type=DLSFLS, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.OpenDistroSecurityFlsDlsIndexSearcherWrapper], Module [type=MULTITENANCY, implementing class=com.amazon.opendistroforelasticsearch.security.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.amazon.opendistroforelasticsearch.security.dlic.rest.api.OpenDistroSecurityRestApiActions], Module [type=AUDITLOG, implementing class=com.amazon.opendistroforelasticsearch.security.auditlog.impl.AuditLogImpl]]
[2019-06-06T16:00:27,208][INFO ][o.e.g.GatewayService     ] [WbbXwRt] recovered [0] indices into cluster_state
[2019-06-06T16:00:27,227][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [WbbXwRt] Will create .opendistro_security index so we can apply default config
[2019-06-06T16:00:27,492][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector OSMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:27,688][WARN ][o.e.t.OutboundHandler    ] [WbbXwRt] send message failed [channel: Netty4TcpChannel{localAddress=/172.21.0.2:33268, remoteAddress=odfe-node1/172.21.0.3:9300}]
javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[?:?]
[2019-06-06T16:00:28,221][INFO ][o.e.c.m.MetaDataCreateIndexService] [WbbXwRt] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1], mappings []
[2019-06-06T16:00:28,325][INFO ][o.e.c.r.a.AllocationService] [WbbXwRt] updating number_of_replicas to [0] for indices [.opendistro_security]
[2019-06-06T16:00:29,504][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector OSMetrics is still in progress, so skipping this Interval
[2019-06-06T16:00:31,267][INFO ][o.e.c.r.a.AllocationService] [WbbXwRt] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.opendistro_security][0]] ...]).
[2019-06-06T16:00:31,324][INFO ][c.a.o.s.s.ConfigHelper   ] [WbbXwRt] Will update 'config' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml
[2019-06-06T16:00:32,396][INFO ][o.e.c.m.MetaDataMappingService] [WbbXwRt] [.opendistro_security/0BCHDZE1T7yhtmoejzC0Ug] create_mapping [security]
[2019-06-06T16:00:34,282][INFO ][c.a.o.s.s.ConfigHelper   ] [WbbXwRt] Will update 'roles' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles.yml
[2019-06-06T16:00:34,427][INFO ][o.e.c.m.MetaDataMappingService] [WbbXwRt] [.opendistro_security/0BCHDZE1T7yhtmoejzC0Ug] update_mapping [security]
[2019-06-06T16:00:34,975][INFO ][c.a.o.s.s.ConfigHelper   ] [WbbXwRt] Will update 'rolesmapping' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/roles_mapping.yml
[2019-06-06T16:00:35,155][INFO ][o.e.c.m.MetaDataMappingService] [WbbXwRt] [.opendistro_security/0BCHDZE1T7yhtmoejzC0Ug] update_mapping [security]
[2019-06-06T16:00:35,535][INFO ][c.a.o.s.s.ConfigHelper   ] [WbbXwRt] Will update 'internalusers' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/internal_users.yml
[2019-06-06T16:00:35,668][INFO ][o.e.c.m.MetaDataMappingService] [WbbXwRt] [.opendistro_security/0BCHDZE1T7yhtmoejzC0Ug] update_mapping [security]
[2019-06-06T16:00:36,293][INFO ][c.a.o.s.s.ConfigHelper   ] [WbbXwRt] Will update 'actiongroups' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/action_groups.yml
[2019-06-06T16:00:36,497][INFO ][o.e.c.m.MetaDataMappingService] [WbbXwRt] [.opendistro_security/0BCHDZE1T7yhtmoejzC0Ug] update_mapping [security]
[2019-06-06T16:00:37,179][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [WbbXwRt] Default config applied
[2019-06-06T16:00:38,436][INFO ][c.a.o.s.c.IndexBaseConfigurationRepository] [WbbXwRt] Node 'WbbXwRt' initialized
[2019-06-06T16:00:39,504][INFO ][o.e.m.j.JvmGcMonitorService] [WbbXwRt] [gc][17] overhead, spent [363ms] collecting in the last [1s]
[2019-06-06T16:07:19,664][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector OSMetrics is still in progress, so skipping this Interval
[2019-06-06T16:08:36,360][INFO ][c.a.o.e.p.c.ScheduledMetricCollectorsExecutor] [WbbXwRt] Collector MasterServiceEventMetrics is still in progress, so skipping this Interval
[2019-06-06T16:08:36,368][WARN ][o.e.m.j.JvmGcMonitorService] [WbbXwRt] [gc][young][489][13] duration [1.5s], collections [1]/[1.9s], total [1.5s]/[3.6s], memory [212.8mb]->[82.1mb]/[495.3mb], all_pools {[young] [131.3mb]->[766.8kb]/[133.1mb]}{[survivor] [7.9mb]->[8mb]/[16.6mb]}{[old] [73.4mb]->[73.4mb]/[345.6mb]}
[2019-06-06T16:08:36,376][WARN ][o.e.m.j.JvmGcMonitorService] [WbbXwRt] [gc][489] overhead, spent [1.5s] collecting in the last [1.9s]

Update 1: It works as expected in my Docker for Windows. It's just the REHL that's acting funny.

alolita commented 4 years ago

Hi @ngtrunghuan did you resolve this issue? If so please share - would be great to add to the ODFE Docker install documentation.

OlivierCuyp commented 4 years ago

@alolita I had the same kind of issue on OSX and Docker. To solve it, I change : ES_JAVA_OPTS=-Xms512m -Xmx512m to ES_JAVA_OPTS=-Xms2g -Xmx2g

Making sure I gave Docker at least 6Gb of RAM.

It seems that 512Mb is not enough for ES version 7 ;)

ngtrunghuan commented 4 years ago

@alolita I actually dropped the issue altogether. All the best!