search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.36k stars 613 forks source link

[Bug] Migration error from TheHive 3.5 to TheHive 4.1.13 #2267

Open aperezdev opened 2 years ago

aperezdev commented 2 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu
OS version (client) Ubuntu
Virtualized Env. True
Dedicated RAM 16 GB
vCPU 8
TheHive version / git hash 4.1.13-1
Package Type DEB
Database Cassandra
Index type Elasticsearch
Attachments storage Local

Problem Description

I'm trying to migrate from TheHive 3.5 to TheHive 4.1 with the following amount of data:

After a few minutes of the migration process, when there are almost 11000 alerts migrated, the migration crush with the following error:

[error] ElasticSearch request failure: POST:/_search/scroll?
StringEntity({"scroll":"60000ms","scroll_id":"************************"},Some(application/json))
=> ElasticError(search_phase_execution_exception,all shards failed,None,None,None,List(ElasticError(search_context_missing_exception,No search context found for id [5159380],None,None,None,null,None,None,None,List()), ElasticError(search_context_missing_exception,No search context found for id [5159381],None,None,None,null,None,None,None,List()), ElasticError(search_context_missing_exception,No search context found for id [5159382],None,None,None,null,None,None,None,List()), ElasticError(search_context_missing_exception,No search context found for id [5159383],None,None,None,null,None,None,None,List()), ElasticError(search_context_missing_exception,No search context found for id [5159384],None,None,None,null,None,None,None,List())),Some(CausedBy(search_context_missing_exception,No search context found for id [5159384],Map())),Some(query),Some(true),List(FailedShard(-1,None,None,Some(ElasticError(search_context_missing_exception,No search context found for id [5159380],None,None,None,null,None,None,None,List()))), FailedShard(-1,None,None,Some(ElasticError(search_context_missing_exception,No search context found for id [5159381],None,None,None,null,None,None,None,List()))), FailedShard(-1,None,None,Some(ElasticError(search_context_missing_exception,No search context found for id [5159382],None,None,None,null,None,None,None,List()))), FailedShard(-1,None,None,Some(ElasticError(search_context_missing_exception,No search context found for id [5159383],None,None,None,null,None,None,None,List()))), FailedShard(-1,None,None,Some(ElasticError(search_context_missing_exception,No search context found for id [5159384],None,None,None,null,None,None,None,List())))))
[warn] Search error
org.thp.scalligraph.SearchError: all shards failed
at org.thp.thehive.migration.th3.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:144)
at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at org.thp.scalligraph.ContextPropagatingDispatcher$$anon$1.$anonfun$execute$2(ContextPropagatingDisptacher.scala:57)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.thp.scalligraph.DiagnosticContext$.$anonfun$withDiagnosticContext$2(ContextPropagatingDisptacher.scala:93)
at org.thp.scalligraph.DiagnosticContext$.saveDiagnosticContext(ContextPropagatingDisptacher.scala:108)
at org.thp.scalligraph.DiagnosticContext$.withDiagnosticContext(ContextPropagatingDisptacher.scala:91)
at org.thp.scalligraph.DiagnosticContext$$anon$2.withContext(ContextPropagatingDisptacher.scala:76)
[error] Migration failed
org.thp.scalligraph.SearchError: Request terminated early or timed out
at org.thp.thehive.migration.th3.SearchWithScroll$$anon$1$$anon$2.$anonfun$onPull$1(DBFind.scala:198)
at org.thp.thehive.migration.th3.SearchWithScroll$$anon$1$$anon$2.$anonfun$onPull$1$adapted(DBFind.scala:178)
at akka.stream.impl.fusing.GraphInterpreter.runAsyncInput(GraphInterpreter.scala:466)
at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:498)
at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:600)
at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:769)
at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:784)
at akka.actor.Actor.aroundReceive(Actor.scala:537)
at akka.actor.Actor.aroundReceive$(Actor.scala:535)
at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:691)

Then, I try to initiate TheHive but after some days, there is a reindex job which never ends making impossible to start TheHive again.

2021-11-25 08:31:50,510 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-6 [|mgmt-5b660023] Reindex job 7249efc2 is running
2021-11-25 08:31:51,510 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-6 [|mgmt-5b660023] Reindex job 7249efc2 is running

What could be the problem of the migration process?

Steps to Reproduce

  1. AWS EC2 instance with TheHive 4.1.13-1 running. Another separate instance with Cassandra running. Connection between Cassandra and TheHive is working fine with SSL and it works fine before starting the migration process
  2. Command to start the migration process: /opt/thehive/bin/migrate --output /etc/thehive/application.conf --main-organisation ORG --es-uri https://******.es.amazonaws.com:443 --es-index the_hive --input thehive3.conf
aperezdev commented 2 years ago

Still same error after testing the migration with TheHive new version 4.1.16. https://blog.strangebee.com/thehive-4-1-16-is-out/