search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.36k stars 613 forks source link

MISP Synchronization completed - but nothing happens #851

Closed Mudamara closed 5 years ago

Mudamara commented 5 years ago

MISP Synchronization completed - but nothing happens

Help requested /Bug

Work Environment

Question Answer
OS version (server) Ubuntu
TheHive version / git hash 3.2.1-1
Package Type DEB, From source (tried both)
Browser type & version Chrome 71.0.3578.98

Problem Description

I have installed HIVE, MISP and Cortex according to the latest documents available on GitHub. I am new-ish to this but not a total buffoon. (we shall see) I have used both the 'DEB' and 'From Source' options to install and have come come to this point both times (suggesting that this is truly a meat-ware problem). No matter what I change, MISP will simply not send Alerts to HIVE.

I have attempted to broaden the filters in various ways.

The MISP instance is the very latest version, and is completely working with a wide variety of ingests and export capabilities with the exception of HIVE.

I have connections between MISP and CORTEX in both directions. I have connections between CORTEX and HIVE.

The most annoying part of this is that the log for HIVE says : update starting, update complete.

Any assistance would be most welcome. This is such a great set of tools. I don't want to be stuck.

Complementary information

latest log output since latest changes to application.conf (file follows

root@misp:/etc/thehive# vim application.conf
root@misp:/etc/thehive# systemctl restart thehive
root@misp:/etc/thehive# tailf /var/log/thehive/application.log
2019-01-21 13:06:43,014 [INFO] from org.elasticsearch.plugins.PluginsService in main - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
2019-01-21 13:06:44,202 [INFO] from io.netty.util.internal.PlatformDependent in main - Your platform does not provide complete low-level API for accessing direct buffers reliably. Unless explicitly requested, heap buffer will always be preferred to avoid potential system instability.
2019-01-21 13:06:45,171 [INFO] from connectors.cortex.services.CortexClient in main - new Cortex(CORTEX, http://172.16.15.51:9001) authentication: connectors.cortex.services.CortexAuthentication$Key
2019-01-21 13:06:45,179 [INFO] from connectors.cortex.services.CortexAnalyzerSrv in main - Search for unfinished job ...
2019-01-21 13:06:45,403 [INFO] from connectors.cortex.services.CortexAnalyzerSrv in application-akka.actor.default-dispatcher-5 - 0 jobs found
2019-01-21 13:06:45,692 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-2 - Update of MISP events is starting ...
2019-01-21 13:06:45,791 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-3 - Misp synchronization completed
2019-01-21 13:06:45,919 [INFO] from play.api.Play in main - Application started (Prod)
2019-01-21 13:06:46,614 [INFO] from play.core.server.AkkaHttpServer in main - Enabling HTTP/2 on Akka HTTP server...
2019-01-21 13:06:46,615 [INFO] from play.core.server.AkkaHttpServer in main - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
2019-01-21 13:07:45,688 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-5 - Update of MISP events is starting ...
2019-01-21 13:07:45,690 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-18 - Misp synchronization completed
2019-01-21 13:08:45,698 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-2 - Update of MISP events is starting ...
2019-01-21 13:08:45,700 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-17 - Misp synchronization completed
2019-01-21 13:09:45,701 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-17 - Update of MISP events is starting ...
2019-01-21 13:09:45,703 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-17 - Misp synchronization completed
2019-01-21 13:10:45,690 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-3 - Update of MISP events is starting ...
2019-01-21 13:10:45,692 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-3 - Misp synchronization completed
2019-01-21 13:11:45,688 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-5 - Update of MISP events is starting ...
2019-01-21 13:11:45,690 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-4 - Misp synchronization completed
2019-01-21 13:12:45,697 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-5 - Update of MISP events is starting ...
2019-01-21 13:12:45,700 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-5 - Misp synchronization completed
2019-01-21 13:13:45,698 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-16 - Update of MISP events is starting ...
2019-01-21 13:13:45,700 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-2 - Misp synchronization completed
2019-01-21 13:14:45,698 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-16 - Update of MISP events is starting ...
2019-01-21 13:14:45,700 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-16 - Misp synchronization completed
2019-01-21 13:15:45,698 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-18 - Update of MISP events is starting ...
2019-01-21 13:15:45,701 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-18 - Misp synchronization completed
2019-01-21 13:16:45,698 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-5 - Update of MISP events is starting ...
2019-01-21 13:16:45,699 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-16 - Misp synchronization completed

Latest version of application.conf

`# Secret Key
# The secret key is used to secure cryptographic functions.
# WARNING: If you deploy your application on several servers, make sure to use the same key.
#play.http.secret.key="***changeme***"
play.http.secret.key="_KEY_"
#Enable HTTPS without reverse proxyy
  # https.port: 9443
   # play.server.https.keyStore {
     #path: "/etc/apache2/keystore.jks"
     #type: "JKS"
     #password: "_PASSWORD_"
  # http.port=disabled
  # session.secure=true
# Elasticsearch
search {
  ## Basic configuration
  # Index name.
  index = the_hive
  # ElasticSearch cluster name.
  cluster = hive
  # ElasticSearch instance address.
  host = ["127.0.0.1:9400"]

  ## Advanced configuration
  # Scroll keepalive.
  keepalive = 1m
  # Scroll page size.
  pagesize = 50
  # Number of shards
  nbshards = 5
  # Number of replicas
  nbreplicas = 1
  # Arbitrary settings
  settings {
  #  # Maximum number of nested fields
  mapping.nested_fields.limit = 100
  }

  ### XPack SSL configuration
  # Username for XPack authentication
  #search.username = ""
  # Password for XPack authentication
  #search.password = ""
  # Enable SSL to connect to ElasticSearch
  search.ssl.enabled = false
  # Path to certificate authority file
  #search.ssl.ca = ""
  # Path to certificate file
  #search.ssl.certificate = ""
  # Path to key file
  #search.ssl.key = ""

  ### SearchGuard configuration
  # Path to JKS file containing client certificate
  #search.guard.keyStore.path = ""
  # Password of the keystore
  #search.guard.keyStore.password = ""
  # Path to JKS file containing certificate authorities
  #search.guard.trustStore.path = ""
  ## Password of the truststore
  #search.guard.trustStore.password = ""
  # Enforce hostname verification
 #search.guard.hostVerification = false
  # If hostname verification is enabled specify if hostname should be resolved
  #search.guard.hostVerificationResolveHostname = false
}
# Datastore
datastore {
  name = data
  # Size of stored data chunks
  chunksize = 50k
  hash {
    # Main hash algorithm /!\ Don't change this value
    main = "SHA-256"
    # Additional hash algorithms (used in attachments)
    extra = ["SHA-1", "MD5"]
  }
  attachment.password = "_PASSWORD_"
}
# Authentication
auth {
        # "provider" parameter contains authentication provider. It can be multi-valued (useful for migration)
        # available auth types are:
        # services.LocalAuthSrv : passwords are stored in user entity (in Elasticsearch). No configuration is required.
        # ad : use ActiveDirectory to authenticate users. Configuration is under "auth.ad" key
        # ldap : use LDAP to authenticate users. Configuration is under "auth.ldap" key
        provider = [local]

  # By default, basic authentication is disabled. You can enable it by setting "method.basic" to true.
  method.basic = false

        ad {
                # The Windows domain name in DNS format. This parameter is required if you do not use
                # 'serverNames' below.
                #domainFQDN = "mydomain.local"

                # Optionally you can specify the host names of the domain controllers instead of using 'domainFQDN
                # above. If this parameter is not set, TheHive uses 'domainFQDN'.
                #serverNames = [ad1.mydomain.local, ad2.mydomain.local]

                # The Windows domain name using short format. This parameter is required.
                #domainName = "MYDOMAIN"

                # If 'true', use SSL to connect to the domain controller.
                #useSSL = true
        }

        ldap {
                # The LDAP server name or address. The port can be specified using the 'host:port'
                # syntax. This parameter is required if you don't use 'serverNames' below.
                #serverName = "ldap.mydomain.local:389"

                # If you have multiple LDAP servers, use the multi-valued setting 'serverNames' instead.
                #serverNames = [ldap1.mydomain.local, ldap2.mydomain.local]

                # Account to use to bind to the LDAP server. This parameter is required.
                #bindDN = "cn=thehive,ou=services,dc=mydomain,dc=local"

                # Password of the binding account. This parameter is required.
                #bindPW = "***secret*password***"

                # Base DN to search users. This parameter is required.
                #baseDN = "ou=users,dc=mydomain,dc=local"
 # Filter to search user in the directory server. Please note that {0} is replaced
                # by the actual user name. This parameter is required.
                #filter = "(cn={0})"

                # If 'true', use SSL to connect to the LDAP directory server.
                #useSSL = true
        }
}

# Maximum time between two requests without requesting authentication
session {
  warning = 5m
  inactivity = 1h
}

# Streaming
stream.longpolling {
  # Maximum time a stream request waits for new element
  refresh = 1m
  # Lifetime of the stream session without request
  cache = 15m
  nextItemMaxWait = 500ms
  globalMaxWait = 1s
}
# Max textual content length
play.http.parser.maxMemoryBuffer= 1M
# Max file size
play.http.parser.maxDiskBuffer = 1G

# Cortex
# TheHive can connect to one or multiple Cortex instances. Give each
# Cortex instance a name and specify the associated URL.
#
# In order to use Cortex, first you need to enable the Cortex module by uncommenting the next line

play.modules.enabled += connectors.cortex.CortexConnector

cortex {
  "CORTEX" {
  # "cortex"{
    url = "http://172.16.15.51:9001"
  #  key = "_KEY_"
    key = "_KEY_"
}
  #  # HTTP client configuration (SSL and proxy)
  # ws {
  #   proxy {}
  #   ssl {}
  # }
  # Check job update time interval
  refreshDelay = 1 minute
  # Maximum number of successive errors before give up
  maxRetryOnError = 3
  # Check remote Cortex status time interval
  statusCheckInterval = 1 minute
}
# MISP
# TheHive can connect to one or multiple MISP instances. Give each MISP
# instance a name and specify the associated Authkey that must  be used
# to poll events, the case template that should be used by default when
# importing events as well as the tags that must be added to cases upon
# import.

# Prior to configuring the integration with a MISP instance, you must
# enable the MISP connector. This will allow you to import events to
# and/or export cases to the MISP instance(s).

play.modules.enabled += connectors.misp.MispConnector

misp {
  "_NAME_" {
  #  # MISP connection configuration requires at least an url and a key. The key must
  #  # be linked with a sync account on MISP.
     url = _"https://MISP"_
   #url = "_http://MISP_IP:PORT_"
   # key = "_ADMIN_KEY_"
   key = "_SYNCUSER_KEY_"
  #  # Name of the case template in TheHive that shall be used to import
  #  # MISP events as cases by default.
    caseTemplate = "MISP-EVENT"
  #  # Optional tags to add to each observable  imported  from  an  event
  #  # available on this instance.
    tags = ["misp"]
  #
  # Truststore to use to validate the X.509 certificate  of  the  MISP
  # instance if the default truststore is not sufficient.
  #ws.ssl.trustManager.stores = [
  #{
  #  type: "JKS"
  #  path: "/path/to/truststore.jks"
  #}
  #]
  #
  #  ## HTTP client configuration (SSL and proxy)
  #  # Truststore to use to validate the X.509 certificate of the MISP
  #  # instance if the default truststore is not sufficient.
  #  # Proxy can also be used
   # ws {
    #  ssl.trustManager.stores = [ {
     #   path = /etc/apache2/keystore.jks
     # } ]
  #    proxy {
  #      host = proxy.mydomain.org
  #      port = 3128
  #    }

  #  ## MISP event filters
  #  # MISP filters is used to exclude events from the import.
  #  # Filter criteria are:
  #  # The number of attribute
    max-attributes = 100
  #  # The size of its JSON representation
    max-size = 500 MiB
  #  # The age of the last publish date
    max-age = 7 days
  #  # Organization and tags
  #  exclusion {
 #    organisation = ["bad organisation", "other organisations"]
  #    tags = ["tag1", "tag2"]
  #}
  whitelist.tags = ["TEST"]
  purpose = ImportAndExport
  }
  # Check remote TheHive status time interval
  statusCheckInterval = 1 minute
  # Interval between consecutive MISP event imports in hours (h) or
  # minutes (m).
  interval = 1m
}
DanteDevil89 commented 5 years ago

Try to change the "max-age = 7 days" and verify if there are events published in the misp server in the last 7 days. Check if you have correctly published an event in MISP.

Mudamara commented 5 years ago

Hi, Sorry for delay in replying. I have changed the "max-age" as suggested, and have used a time period long enough to incorporate every single event in MISP and have used more sensible time periods like 30/60/90 days etc. I have many correctly published events and still nothing.

Perhaps put this question on hold/closed for now, I plan to do a ground up rebuild of everything to see if there is something simple I have missed along the way.

Wish me luck and thanks for you suggestion. I am sure Ill be back in contact soon!