search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.39k stars 617 forks source link

Misp synchronization failed #740

Open garanews opened 6 years ago

garanews commented 6 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu
TheHive version 3.0.10
MISP version 2.4.95
Package Type DEB

Problem Description

Issue started Sept 20th.

2018-09-20 09:23:05,025 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-4844 - Misp synchronization failed
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting comma to separate Array entries
 at [Source: java.io.StringReader@2cebc402; line: 1, column: 9824893]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipComma(ReaderBasedJsonParser.java:2285)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:674)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:179)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:128)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:123)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2115)
        at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:235)
        at play.api.libs.json.StaticBinding$.parseJsValue(StaticBinding.scala:16)

Workaround when it happens is comment misp from thehive/application.conf, restart thehive, uncomment misp thehive/application.conf, from and restart thehive again.

To-om commented 6 years ago

Hi @garanews,

This error means that MISP server doesn't return valid response format. TheHive expects a JSON.

Is it possible for you to capture communication between TheHive and MISP (if ssl is disabled on MISP)?

You can also enable MISP debug by adding <logger name="connectors.misp" level="DEBUG" /> in /etc/thehive/logback.xml file and restart TheHive.

garanews commented 5 years ago

It is possible that response is truncated because too big? Enabling the debug it seems that the event causing the issue has more than 60k attributes. The entire json is around 20MB, and if I try to pull from MISP I am able to download without issues.

smclinden commented 5 years ago

I'm seeing something slightly different:

018-10-03 13:26:41,941 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-17 - Update of MISP even
ts is starting ...
2018-10-03 13:26:42,077 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-16 - Synchronize MISP Mo
lina_MISP from Some(Fri Sep 14 11:22:35 PDT 2018)
2018-10-03 13:26:42,080 [INFO] from play.api.Play in main - Application started (Prod)
2018-10-03 13:26:42,743 [INFO] from play.core.server.AkkaHttpServer in main - Enabling HTTP/2 on Akka HTTP server...
2018-10-03 13:26:42,747 [INFO] from play.core.server.AkkaHttpServer in main - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
2018-10-03 13:26:42,956 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-19 - Misp synchronizatio
n failed
play.api.libs.json.JsResultException: JsResultException(errors:List(((0),List(JsonValidationError(List(null is not an object),Wrapped
Array()))), ((1),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((2),List(JsonValidationError(List(null is n
ot an object),WrappedArray()))), ((3),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((4),List(JsonValidatio
nError(List(null is not an object),WrappedArray()))), ((5),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((
6),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((7),List(JsonValidationError(List(null is not an object),
WrappedArray()))), ((8),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((9),List(JsonValidationError(List(nu
ll is not an object),WrappedArray()))), ((10),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((11),List(Json
ValidationError(List(null is not an object),WrappedArray()))), ((12),List(JsonValidationError(List(null is not an object),WrappedArra
y()))), ((13),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((14),List(JsonValidationError(List(null is not
 an object),WrappedArray()))), ((15),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((16),List(JsonValidatio
nError(List(null is not an object),WrappedArray()))), ((17),List(JsonValidationError(List(null is not an object),WrappedArray()))), (
(18),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((19),List(JsonValidationError(List(null is not an objec
t),WrappedArray()))), ((20),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((21),List(JsonValidationError(Li
st(null is not an object),WrappedArray()))), ((22),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((23),List
(JsonValidationError(List(null is not an object),WrappedArray()))), ((24),List(JsonValidationError(List(null is not an object),Wrappe
dArray()))), ((25),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((26),List(JsonValidationError(List(null i
s not an object),WrappedArray()))), ((27),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((28),List(JsonVali
dationError(List(null is not an object),WrappedArray()))), ((29),List(JsonValidationError(List(null is not an object),WrappedArray())
)), ((30),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((31),List(JsonValidationError(List(null is not an
object),WrappedArray()))), ((32),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((33),List(JsonValidationErr
or(List(null is not an object),WrappedArray()))), ((34),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((35)
,List(JsonValidationError(List(null is not an object),WrappedArray()))), ((36),List(JsonValidationError(List(null is not an object),W
rappedArray()))), ((37),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((38),List(JsonValidationError(List(n
ull is not an object),WrappedArray()))), ((39),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((40),List(Jso
nValidationError(List(null is not an object),WrappedArray()))), ((41),List(JsonValidationError(List(null is not an object),WrappedArr
ay()))), ((42),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((43),List(JsonValidationError(List(null is no
t an object),WrappedArray()))), ((44),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((45),List(JsonValidati
onError(List(null is not an object),WrappedArray()))), ((46),List(JsonValidationError(List(null is not an object),WrappedArray()))),
((47),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((48),List(JsonValidationError(List(null is not an obje
ct),WrappedArray()))), ((49),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((50),List(JsonValidationError(L
ist(null is not an object),WrappedArray()))), ((51),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((52),Lis
t(JsonValidationError(List(null is not an object),WrappedArray()))), ((53),List(JsonValidationError(List(null is not an object),Wrapp
edArray()))), ((54),List(JsonValidationError(List(null is not an object),WrappedArray()))), ((55),List(JsonValidationError(List(null
is not an object),WrappedArray())))))
        at play.api.libs.json.JsReadable.$anonfun$as$2(JsReadable.scala:25)
        at play.api.libs.json.JsError.fold(JsResult.scala:56)
        at play.api.libs.json.JsReadable.as(JsReadable.scala:24)
        at play.api.libs.json.JsReadable.as$(JsReadable.scala:23)
        at play.api.libs.json.JsArray.as(JsValue.scala:91)
        at connectors.misp.MispSrv.$anonfun$getAttributesFromMisp$10(MispSrv.scala:137)
        at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:241)
        at scala.collection.immutable.List.foreach(List.scala:389)
        at scala.collection.TraversableLike.flatMap(TraversableLike.scala:241)
        at scala.collection.TraversableLike.flatMap$(TraversableLike.scala:238)
        at scala.collection.immutable.List.flatMap(List.scala:352)
        at connectors.misp.MispSrv.$anonfun$getAttributesFromMisp$8(MispSrv.scala:137)
        at scala.util.Success.$anonfun$map$1(Try.scala:251)
        at scala.util.Success.map(Try.scala:209)
        at scala.concurrent.Future.$anonfun$map$1(Future.scala:288)
        at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:29)
        at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:29)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:60)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:91)
        at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:81)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:91)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:43)
        at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
smclinden commented 5 years ago

This seems to have been fixed with the latest "git pull" of MISP.

To-om commented 5 years ago

@garanews the entire json you download from MISP is it valid ? The "Unexpected character" error suggests that this is not a truncated data.

garanews commented 5 years ago

Downloading manually from MISP gui I get full and correct json.

To-om commented 5 years ago

Have you tried to enable debug ? The invalid Json may not be the list of events (TheHive download other elements of MISP)

garanews commented 5 years ago

with debug on just couple of rows appear but nothing interesting except the event id I already reported:

2018-10-09 14:00:56,255 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-4 - Update of MISP events is starting ...
2018-10-09 14:00:56,364 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-2 - Synchronize MISP MISP-1 from Some(Tue Oct 09 11:35:24 UTC 2018)
**2018-10-09 14:00:56,366 [DEBUG] from connectors.misp.MispSrv in application-akka.actor.default-dispatcher-2 - Get MISP events from Tue Oct 09 11:35:24 UTC 2018**
2018-10-09 14:00:56,371 [INFO] from play.api.Play in main - Application started (Prod)
2018-10-09 14:00:56,816 [INFO] from play.core.server.AkkaHttpServer in main - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
**2018-10-09 14:01:01,065 [DEBUG] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-8 - getting MISP event MISP-MISP1:3531**
2018-10-09 14:01:10,289 [INFO] from connectors.misp.MispSynchro in application-akka.actor.default-dispatcher-6 - Misp synchronization failed
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('{' (code 123)): was expecting comma to separate Array entries
 at [Source: java.io.StringReader@4604748; line: 1, column: 9824880]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1702)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:558)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:456)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipComma(ReaderBasedJsonParser.java:2285)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:674)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:179)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:128)
        at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:123)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3786)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2115)
        at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:235)
        at play.api.libs.json.StaticBinding$.parseJsValue(StaticBinding.scala:16)
        at play.api.libs.json.Json$.parse(Json.scala:171)
        at connectors.misp.MispSrv.$anonfun$getAttributesFromMisp$4(MispSrv.scala:120)
        at scala.util.Success.$anonfun$map$1(Try.scala:251)
        at scala.util.Success.map(Try.scala:209)
cyhook commented 5 years ago

Is there a solution for this? I am getting the same error on docker