search

TheHive-Project / Cortex

Cortex: a Powerful Observable Analysis and Active Response Engine
https://strangebee.com/cortex/
GNU Affero General Public License v3.0
1.33k stars 228 forks source link

Cortex and MISP unclear and error-loop #29

Closed crackytsi closed 7 years ago

crackytsi commented 7 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Debian 8
Cortex version / git hash 1.1.1-2
Package Type Debian Package

Problem Description

Hello, Thanks a lot for your really, really good work!!! Sorry maybe its my fault, but I don't have any further idear, so I use this way to adress it:

  1. My Cortex Config loops persmantenly this messages: 
    
May 19 21:02:46 debian-8-user cortex[19470]: import misp_modules
May 19 21:02:46 debian-8-user cortex[19470]: ImportError: No module named 'misp_modules'
May 19 21:02:46 debian-8-user cortex[19470]: [#033[37minfo#033[0m] application - GET /api/analyzer returned 500
May 19 21:02:46 debian-8-user cortex[19470]: java.lang.RuntimeException: Nonzero exit value: 1
May 19 21:02:46 debian-8-user cortex[19470]: at scala.sys.package$.error(package.scala:27)
May 19 21:02:46 debian-8-user cortex[19470]: at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:132)
May 19 21:02:46 debian-8-user cortex[19470]: at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:102)
May 19 21:02:46 debian-8-user cortex[19470]: at services.MispSrv.list$lzycompute(MispSrv.scala:46)
May 19 21:02:46 debian-8-user cortex[19470]: at services.MispSrv.list(MispSrv.scala:45)
May 19 21:02:46 debian-8-user cortex[19470]: at services.AnalyzerSrv.list(AnalyzerSrv.scala:18)
May 19 21:02:46 debian-8-user cortex[19470]: at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:19)
May 19 21:02:46 debian-8-user cortex[19470]: at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:18)
May 19 21:02:46 debian-8-user cortex[19470]: at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
May 19 21:02:46 debian-8-user cortex[19470]: at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
May 19 21:02:47 debian-8-user cortex[19470]: Traceback (most recent call last):
May 19 21:02:47 debian-8-user cortex[19470]: File "/opt/cortex/contrib/misp-modules-loader.py", line 10, in <module>
2. Why are there now 2 different stancas? What do they affect?

analyzer { path = "/opt/Cortex-Analyzers/analyzers" config { ... } 


AND

misp.modules { enabled = true

config { ... }



3. Is it possible provide one "full" config file containing all possible parameters (as remark?)
   I'm not sure if the old configuration of MISP (as source) is still correct:
    MISP {
            url="https://server"
            key="mykey"
            certpath=["/etc/ssl/private/misp.local.crt", ""]
            name="instance-1"
    }
saadkadhi commented 7 years ago

Hi @crackytsi,

The latest version of Cortex as of this writing (v 1.1.1) lets you invoke MISP expansion modules if you'd like, as described in the MISP Integration Guide. That is the reason why there is now a separate stanza for them. We prefer to keep the configuration of Cortex native analyzers separate from the configuration of the MISP expansion modules.

I successfully reproduced your problem. It is due to the fact that the MISP modules are enabled by default and Cortex is looking for them. Please disable the MISP modules in application.conf:

misp.modules {
  enabled = false
...

And restart Cortex. This should fix your problem temporarily. Now that we are aware of this issue, we are going to look for a permanent solution very soon.

Thank you for your nice comments about our work. We should held ourselves to better QA standards though to prevent such errors from happening in the first place. This will be our priority in the next few weeks.

crackytsi commented 7 years ago

Thank you so much. I understand it now :)

crackytsi commented 7 years ago

Hi, I installed the plugins from MISP as described. Never the less I'm still faced with issues (also in a loop): I have no idear where uwhois is defined (a grep did not help me). Any hint?


2017-05-22 10:36:33,228 [INFO] from application in application-akka.actor.default-dispatcher-5 - GET /api/analyzer returned 500
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'uwhois': was expecting ('true', 'false' or 'null')
 at [Source: uwhois module not installed.
["wiki", "asn_history", "dns", "sourcecache", "eupi", "whois", "circl_passivedns", "virustotal", "cve", "shodan", "circl_passivessl", "geoip_country", "ipasn", "passivetotal", "domaintools", "iprep", "reversedns", "countrycode", "vmray_submit", "threatminer"]
; line: 1, column: 7]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2749)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1820)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:708)
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3765)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)
        at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:238)
        at play.api.libs.json.Json$.parse(Json.scala:21)
        at services.MispSrv.list$lzycompute(MispSrv.scala:46)
        at services.MispSrv.list(MispSrv.scala:45)
        at services.AnalyzerSrv.list(AnalyzerSrv.scala:18)
        at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:19)
        at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:18)
        at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
        at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
        at play.api.mvc.Action$.invokeBlock(Action.scala:498)
        at play.api.mvc.Action$.invokeBlock(Action.scala:495)
        at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:111)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:110)
        at scala.Option.map(Option.scala:146)
        at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
        at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
nadouani commented 7 years ago

This uwhois is a dependency of one of the misp-modules

crackytsi commented 7 years ago

Hmmm. But I disabled all except dns lookup...

nadouani commented 7 years ago

Well, When misp-modules is enabled in Cortex, the latter will try to load the list of all misp-modules available (undependently from what modules you have configured)

When listing the misp modules, the operation fail because of the missing uwhois package.

The issue is that this package is not available on PIP (it's a fork a another uwhoisd project...)

To get the uwhois installed, make sure your pip is up to date:

sudo pip install pip --upgrade

and then run

sudo pip install 'git+https://github.com/Rafiot/uwhoisd.git@testing#egg=uwhois&subdirectory=client'
crackytsi commented 7 years ago

Thanks, I got it. At the time time you wrote this, I finally found this commit: https://github.com/Rafiot/uwhoisd/commit/3f16d4261f3cd6ffe8876b4a68d3733e270dc4e2 :)