Open Deastrom opened 7 years ago
Hi, did you get this working? Did you modify docker-compose.yml at all?
I interpret the installation docs the same as you; either use docker-compose or manually install and configure Elasticsearch, but I just copied the docker-compose.yml file to an empty directory and ran 'sudo docker-compose up' and I don't get the same error, it seems to work fine.
Can you provide the exact steps you went through and the full output?
I ended up using apt-get
On Tue, Sep 5, 2017, 8:42 PM npratley notifications@github.com wrote:
Hi, did you get this working? Did you modify docker-compose.yml at all?
I interpret the installation docs the same as you; either use docker-compose or manually install and configure Elasticsearch, but I just copied the docker-compose.yml file to an empty directory and ran 'sudo docker-compose up' and I don't get the same error, it seems to work fine.
Can you provide the exact steps you went through and the full output?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CERT-BDF/TheHiveDocs/issues/8#issuecomment-327350036, or mute the thread https://github.com/notifications/unsubscribe-auth/AY8JrBr2qiX2xdc-S4lAEJaSmeQ-lb-Cks5sffh7gaJpZM4O8ZxY .
Ok, so for those who don't want to follow @Deastrom in laboriously setting up elasticsearch the hard way just to see if The Hive and it's associated Cortex stuff can do what you need it to do (before putting it into production), you may find buried in the docker messages something about your virtual memory stuff being too low.
vm.max_map_count tells you it doesn't have enough memory.
If you have this error, you may also realize that when you start up docker-compose, it doesn't ask you to setup an account when you try and access The Hive with your browser, and just asks for your Username and PW (Then says the elasticsearch cluster isn't reachable).
The fix is simple. Either run this in the command line (tested and works on CENTOS 7),
sysctl -w vm.max_map_count=262144
Or put vm.max_map_count=262144 in /etc/sysctl.conf then reboot!
NOTE: Also, the documentation for installing docker IS telling you it's one or the other. If you use docker-compose, you do NOT have to build out an elasticsearch image, as the docker-compose.yml file provides that too.
I did get it working using that exact command. You'll also need at least 2 gb ram to your docker for Windows vm (if that's what you're using). The hardest lesson in this process was to take out the ip addresses to elastic and cortex instances in your conf file. The hive docker image is designed to look for these services by name and connect automatically.
version: "3" services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2 environment:
Thanks for adding that file! That looks great, and cleared up a few other things I was looking into!
Error:
[info] o.e.ErrorHandler - POST /api/login returned 500 org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{PBRk09qYSgumacUfFfRNVg}{0.0.0.0}{0.0.0.0:9300}] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1256) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) at com.sksamuel.elastic4s.admin.IndexAdminExecutables$IndexExistsDefinitionExecutable$.apply(IndexAdminExecutables.scala:53) at com.sksamuel.elastic4s.admin.IndexAdminExecutables$IndexExistsDefinitionExecutable$.apply(IndexAdminExecutables.scala:50)
Docker compose: version: "3" services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:5.5.2 ports:
image: thehiveproject/cortex:latest ports:
thehive:
build: . depends_on:
Tried with and without this Dockerfile: FROM thehiveproject/thehive:latest USER root RUN chgrp -R 0 /opt USER daemon
My config in thehive/conf/application.conf:
play.http.secret.key="VLROlUsB5yVvZFBGM3KRRHO4ihFaat8wpNfwjsQWzVmcL6c8jspbb2pTL6SvhyLT"
search {
index = the_hive
cluster = hive
host = ["0.0.0.0:9300"]
keepalive = 1m
pagesize = 50
nbshards = 5
nbreplicas = 1
settings {
mapping.nested_fields.limit = 100
}
search.ssl.enabled = false client.transport.sniff = true
}
auth {
# available auth types are:
# services.LocalAuthSrv : passwords are stored in user entity (in Elasticsearch). No configuration is required.
# ad : use ActiveDirectory to authenticate users. Configuration is under "auth.ad" key
# ldap : use LDAP to authenticate users. Configuration is under "auth.ldap" key
provider = [local]
method.basic = false
ad {
# The name of the Microsoft Windows domain using the DNS format. This parameter is required.
#domainFQDN = "mydomain.local"
# Optionally you can specify the host names of the domain controllers. If not set, TheHive uses "domainFQDN".
#serverNames = [ad1.mydomain.local, ad2.mydomain.local]
# The Microsoft Windows domain name using the short format. This parameter is required.
#domainName = "MYDOMAIN"
# Use SSL to connect to the domain controller(s).
#useSSL = true
}
ldap {
#serverName = "ldap.mydomain.local:389"
# If you have multiple ldap servers, use the multi-valued settings.
#serverNames = [ldap1.mydomain.local, ldap2.mydomain.local]
# Use SSL to connect to directory server
#useSSL = true
# Account to use to bind on LDAP server. This parameter is required.
#bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local"
# Password of the binding account. This parameter is required.
#bindPW = "***secret*password***"
# Base DN to search users. This parameter is required.
#baseDN = "ou=users,dc=mydomain,dc=local"
# Filter to search user {0} is replaced by user name. This parameter is required.
#filter = "(cn={0})"
}
}
session { warning = 5m inactivity = 1h }
stream.longpolling {
refresh = 1m
cache = 15m nextItemMaxWait = 500ms globalMaxWait = 1s }
play.http.parser.maxMemoryBuffer=1M
play.http.parser.maxDiskBuffer=1G
#
play.modules.enabled += connectors.cortex.CortexConnector cortex { "CORTEX-SERVER-ID" {
url = "http://cortex:9001"
# Key of the Cortex user, mandatory for Cortex 2
key = "SqQ28IbY11D4t0QzlbP3VpZnkyNIGlAd"
}
}
datastore { name = data
chunksize = 50k hash {
main = "SHA-256"
# Additional hash algorithms (used in attachments)
extra = ["SHA-1", "MD5"]
} attachment.password = "malware" } webhooks { myLocalWebHook { url = "http://159.203.179.39:5000" } }
What I know: elasticsearch is define in /etc/hosts cxe@thehive:~/TheHive$ curl 'elasticsearch:9200/_cat/indices?v' health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open .monitoring-es-6-2018.12.22 _F-w6qd1S1S6ueXmnYNr5w 1 1 46348 100 30.3mb 30.3mb yellow open .monitoring-alerts-6 DsCpHe7xQRW9SzpzrH4e1w 1 1 1 0 12.4kb 12.4kb yellow open .watcher-history-3-2018.12.22 O6dPES_GTS-mhZ2jx9tfRg 1 1 3835 0 5.7mb 5.7mb yellow open .triggered_watches AhcQJ8skS_Wf_MluhDH6Qw 1 1 0 0 9.5kb 9.5kb yellow open cortex_2 T_2-kX-QRvmcr39xRtoblw 5 1 2 0 12kb 12kb yellow open .watches kWm96sv1Tx6sB2UnbV_c7Q 1 1 4 0 22.8kb 22.8kb
This also works from inside the "thehive" docker container
No index created for thehive
cxe@thehive:~/TheHive$ curl 'localhost:9200' { "name" : "RRI823_", "cluster_name" : "hive", "cluster_uuid" : "mVwIDm4kRhqrPXvvysBFww", "version" : { "number" : "5.5.2", "build_hash" : "b2f0c09", "build_date" : "2017-08-14T12:33:14.154Z", "build_snapshot" : false, "lucene_version" : "6.6.0" }, "tagline" : "You Know, for Search" }
Able to log into Cortex Not able to log into thehive
When I do not use my config file I am able to log into thehive.
I got it figured out.
in application.conf in the search section changed from Address of the Elasticsearch instance host = ["0.0.0.0:9300"] to host = ["elasticsearch:9300"]
https://github.com/CERT-BDF/TheHiveDocs/blob/master/installation/docker-guide.md
The instructions in the above link reads as though you either need to use docker-compose OR manually install and configure ElasticSearch.
I'm pretty new to this so I went the docker-compose route and the instructions give the the following error...
thehive_1 | [error] a.a.OneForOneStrategy - None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{127.0.0.1:9200}] thehive_1 | org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{127.0.0.1:9200}] thehive_1 | at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290) thehive_1 | at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207) thehive_1 | at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55) thehive_1 | at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288) thehive_1 | at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359) thehive_1 | at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:582) thehive_1 | at com.sksamuel.elastic4s.SearchDsl$SearchDefinitionExecutable$$anonfun$apply$1.apply(SearchDsl.scala:40) thehive_1 | at com.sksamuel.elastic4s.SearchDsl$SearchDefinitionExecutable$$anonfun$apply$1.apply(SearchDsl.scala:40) thehive_1 | at com.sksamuel.elastic4s.Executable$class.injectFutureAndMap(Executable.scala:21) thehive_1 | at com.sksamuel.elastic4s.SearchDsl$SearchDefinitionExecutable$.injectFutureAndMap(SearchDsl.scala:37)
The instructions may need to be modified to be more clear.