search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.29k stars 609 forks source link

Elasticsearch crashed after docker-compose up #612

Closed ghost closed 6 years ago

ghost commented 6 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu Server 16.04
OS version (client) Ubuntu Server 16.04
TheHive version / git hash certbdf/thehive:latest
Package Type Docker
Browser type & version N/A

Problem Description

Elasticsearch failed to create node environment, so it exits itself with code 1

Steps to Reproduce

  1. populate docker-compose.yml with code from https://github.com/TheHive-Project/TheHiveDocs/blob/master/installation/install-guide.md#docker and add 
    volumes:
- /usr/share/elasticsearch/data:/usr/share/elasticsearch/data
  2. run sudo docker-compose up

Possible Solutions

N/A

Complementary information

Whole log after executing sudo docker-compose up:

coky@ubuntu:~$ sudo docker-compose up
[sudo] password for coky:
Creating network "coky_default" with the default driver
Pulling elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:5.6.0)...
5.6.0: Pulling from elasticsearch/elasticsearch
364f9b7c969a: Pull complete
ddbb62cd8870: Pull complete
886fc8178f2e: Pull complete
995cd25c1497: Pull complete
9ba1f66bf70b: Pull complete
146f071c5d90: Pull complete
5083971e38f1: Pull complete
fe9db573c6ba: Pull complete
5a0d8f0e6ea8: Pull complete
5dbdb0387426: Pull complete
Digest: sha256:f95e7d4256197a9bb866b166d9ad37963dc7c5764d6ae6400e551f4987a659d7
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch                                                                                        :5.6.0
Pulling cortex (certbdf/cortex:latest)...
latest: Pulling from certbdf/cortex
c73ab1c6897b: Pull complete
1ab373b3deae: Pull complete
b542772b4177: Pull complete
57c8de432dbe: Pull complete
da44f64ae999: Pull complete
0bbc7b377a91: Pull complete
1b6c70b3786f: Pull complete
d9bbcf733166: Pull complete
b1d3e8de8ec6: Pull complete
216fbb82b452: Pull complete
2e729c77756b: Pull complete
b06698504c66: Pull complete
51fe92556313: Pull complete
2231b5598123: Pull complete
946f8497d213: Pull complete
3936e2d85bd7: Pull complete
1a03272a474c: Pull complete
Digest: sha256:e432ebd25c446bf664b0106e0557ee0a9d5ba702db77b515bae51ff75e305695
Status: Downloaded newer image for certbdf/cortex:latest
Pulling thehive (certbdf/thehive:latest)...
latest: Pulling from certbdf/thehive
c73ab1c6897b: Already exists
1ab373b3deae: Already exists
b542772b4177: Already exists
57c8de432dbe: Already exists
da44f64ae999: Already exists
0bbc7b377a91: Already exists
1b6c70b3786f: Already exists
d9bbcf733166: Already exists
b1d3e8de8ec6: Already exists
490b50e44206: Pull complete
317a2a489905: Pull complete
8c53b5f8fc45: Pull complete
74d26cb63690: Pull complete
cb8ed405e529: Pull complete
10d6e008d0f1: Pull complete
112af49375f0: Pull complete
Digest: sha256:c33666f60318de4a0aaf5a7401e2703e6e22ef01c3505177bacfb5b36557e0f9
Creating coky_elasticsearch_1 ... done
Creating coky_cortex_1        ... done
Creating coky_cortex_1        ...
Creating coky_thehive_1       ... done
Attaching to coky_elasticsearch_1, coky_cortex_1, coky_thehive_1
cortex_1         | Using secret: y9ZQ8waegjSAYUhdAixORspwXhyYFZ2e0Ce2ZfisSJjlvDKyIvsanDK1Bn537W5U
cortex_1         | Using elasticsearch host: ["172.18.0.2:9300"]
thehive_1        | Using secret: D1Z7LGexPWEZDl1b38OybLNww16lumYFxsk8PIrYxfiwpouAgb0xu5ycXZyIEhmy
thehive_1        | Using elasticsearch host: ["172.18.0.2:9300"]
thehive_1        | Add Cortex cortex1: http://172.18.0.3:9000
cortex_1         | [info] o.r.Reflections - Reflections took 568 ms to scan 2 urls, producing 95 keys and 871 values
cortex_1         | [info] module - Loading model class org.thp.cortex.models.ArtifactModel
cortex_1         | [info] module - Loading model class org.elastic4play.services.AttachmentModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.OrganizationModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.JobModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.UserModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.ReportModel
cortex_1         | [info] module - Loading model class org.elastic4play.services.DBListModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.AnalyzerConfigModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.AuditModel
cortex_1         | [info] module - Loading model class org.thp.cortex.models.AnalyzerModel
elasticsearch_1  | [2018-06-18T09:15:52,183][INFO ][o.e.n.Node               ] [] initializing ...
elasticsearch_1  | [2018-06-18T09:15:52,285][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
elasticsearch_1  | org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
elasticsearch_1  |      at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  | Caused by: java.lang.IllegalStateException: Failed to create node environment
elasticsearch_1  |      at org.elasticsearch.node.Node.<init>(Node.java:268) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      ... 6 more
elasticsearch_1  | Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes
elasticsearch_1  |      at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
elasticsearch_1  |      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
elasticsearch_1  |      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
elasticsearch_1  |      at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:?]
elasticsearch_1  |      at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_141]
elasticsearch_1  |      at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_141]
elasticsearch_1  |      at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_141]
elasticsearch_1  |      at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:221) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.node.Node.<init>(Node.java:265) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.0.jar:5.6.0]
elasticsearch_1  |      ... 6 more
coky_elasticsearch_1 exited with code 1
thehive_1        | [info] o.r.Reflections - Reflections took 460 ms to scan 5 urls, producing 116 keys and 1174 values
thehive_1        | [info] module - Loading model class models.TaskModel
thehive_1        | [info] module - Loading model class org.elastic4play.services.AttachmentModel
thehive_1        | [info] module - Loading model class models.CaseModel
thehive_1        | [info] module - Loading model class models.ArtifactModel
thehive_1        | [info] module - Loading model class models.AuditModel
thehive_1        | [info] module - Loading model class models.AlertModel
thehive_1        | [info] module - Loading model class models.CaseTemplateModel
thehive_1        | [info] module - Loading model class connectors.cortex.models.ReportTemplateModel
thehive_1        | [info] module - Loading model class models.UserModel
thehive_1        | [info] module - Loading model class models.LogModel
thehive_1        | [info] module - Loading model class org.elastic4play.services.DBListModel
thehive_1        | [info] module - Loading model class models.DashboardModel
thehive_1        | [info] module - Loading model class connectors.cortex.models.JobModel
cortex_1         | [info] a.e.s.Slf4jLogger - Slf4jLogger started
cortex_1         | ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
thehive_1        | [info] a.e.s.Slf4jLogger - Slf4jLogger started
thehive_1        | [info] o.e.p.PluginsService - no modules loaded
thehive_1        | [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
thehive_1        | [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
thehive_1        | [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
thehive_1        | [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
thehive_1        | [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.transport.Netty3Plugin]
thehive_1        | [info] o.e.p.PluginsService - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
thehive_1        | [info] i.n.u.i.PlatformDependent - Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
cortex_1         | [info] o.t.c.s.AnalyzerSrv - New analyzer list:
cortex_1         |
cortex_1         |      CuckooSandbox_Url_Analysis 1.0
cortex_1         |      VirusTotal_GetReport 3.0
cortex_1         |      Shodan_Search 1.0
cortex_1         |      Cymon_Check_IP 2.0
cortex_1         |      PassiveTotal_Osint 2.0
cortex_1         |      CIRCLPassiveDNS 2.0
cortex_1         |      Shodan_Host 1.0
cortex_1         |      TorProject 1.0
cortex_1         |      CIRCLPassiveSSL 2.0
cortex_1         |      Yara 2.0
cortex_1         |      Onyphe_Threats 1.0
cortex_1         |      EmergingThreats_DomainInfo 1.0
cortex_1         |      DNSDB_DomainName 2.0
cortex_1         |      Onyphe_Forward 1.0
cortex_1         |      PhishTank_CheckURL 2.1
cortex_1         |      OTXQuery 2.0
cortex_1         |      C1fApp 1.0
cortex_1         |      Nessus 2.0
cortex_1         |      MISP 2.0
cortex_1         |      JoeSandbox_File_Analysis_Inet 2.0
cortex_1         |      Virusshare 2.0
cortex_1         |      CuckooSandbox_File_Analysis_Inet 1.0
cortex_1         |      Onyphe_Ports 1.0
cortex_1         |      DomainTools_ReverseIP 2.0
cortex_1         |      Yeti 1.0
cortex_1         |      Robtex_Forward_PDNS_Query 1.0
cortex_1         |      VMRay 2.0
cortex_1         |      Abuse_Finder 2.0
cortex_1         |      VirusTotal_Scan 3.0
cortex_1         |      EmergingThreats_IPInfo 1.0
cortex_1         |      Fortiguard_URLCategory 2.0
cortex_1         |      PassiveTotal_Whois_Details 2.0
cortex_1         |      DomainTools_WhoisLookup 2.0
cortex_1         |      PassiveTotal_Malware 2.0
cortex_1         |      DomainTools_ReverseNameServer 2.0
cortex_1         |      DNSDB_IPHistory 2.0
cortex_1         |      WOT_Lookup 1.0
cortex_1         |      GoogleSafebrowsing 2.0
cortex_1         |      PassiveTotal_Enrichment 2.0
cortex_1         |      PayloadSecurity_File_Analysis 1.0
cortex_1         |      PassiveTotal_Unique_Resolutions 2.0
cortex_1         |      DomainTools_ReverseWhois 2.0
cortex_1         |      EmergingThreats_MalwareInfo 1.0
cortex_1         |      SinkDB 1.0
cortex_1         |      DomainTools_WhoisHistory 2.0
cortex_1         |      PassiveTotal_Ssl_Certificate_History 2.0
cortex_1         |      Malpedia 1.0
cortex_1         |      Msg_Parser 2.0
cortex_1         |      CERTatPassiveDNS 2.0
cortex_1         |      PayloadSecurity_Url_Analysis 1.0
cortex_1         |      Hipposcore 2.0
cortex_1         |      JoeSandbox_File_Analysis_Noinet 2.0
cortex_1         |      TorBlutmagie 1.0
cortex_1         |      FireHOLBlocklists 2.0
cortex_1         |      Robtex_Reverse_PDNS_Query 1.0
cortex_1         |      PassiveTotal_Ssl_Certificate_Details 2.0
cortex_1         |      Onyphe_Reverse 1.0
cortex_1         |      DNSDB_NameHistory 2.0
cortex_1         |      Onyphe_Geolocate 1.0
cortex_1         |      PhishingInitiative_Lookup 2.0
cortex_1         |      File_Info 2.0
cortex_1         |      DomainTools_WhoisLookup_IP 2.0
cortex_1         |      JoeSandbox_Url_Analysis 2.0
cortex_1         |      Censys 1.0
cortex_1         |      Robtex_IP_Query 1.0
cortex_1         |      HippoMore 2.0
cortex_1         |      HybridAnalysis_GetReport 1.0
cortex_1         |      PassiveTotal_Passive_Dns 2.0
cortex_1         |      MISPWarningLists 1.0
cortex_1         |      MaxMind_GeoIP 3.0
cortex_1         |
cortex_1         | [info] play.api.Play - Application started (Prod)
thehive_1        | [info] c.c.s.CortexClient - new Cortex(cortex1, http://172.18.0.3:9000) authentication: no
thehive_1        | [info] c.c.s.CortexSrv - Search for unfinished job ...
cortex_1         | [info] p.c.s.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9001
thehive_1        | [info] play.api.Play - Application started (Prod)
thehive_1        | [info] p.c.s.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9000
To-om commented 6 years ago

The directory /usr/share/elasticsearch/data in your host must exist and must be owned by 1000:1000. Run :

mkdir /usr/share/elasticsearch/data
chown 1000:1000 /usr/share/elasticsearch/data

I close the issue, reopen it if this doesn't solve your problem.

exenin commented 5 years ago

The directory /usr/share/elasticsearch/data in your host must exist and must be owned by 1000:1000. Run :

mkdir /usr/share/elasticsearch/data
chown 1000:1000 /usr/share/elasticsearch/data

I close the issue, reopen it if this doesn't solve your problem.

The above issue and Host VM needing more virtual memory map was my issue, thanks !

chown -R /var/lib/elasticsearch/ and sysctl -w vm.max_map_count=262144

https://www.elastic.co/guide/en/elasticsearch/reference/current/vm-max-map-count.html