If I set the parameter 'db.janusgraph.forceDropAndRebuildIndex: true' when using Elasticsearch as index backend, after a while lots of entities are marked as duplicated and data from observables are lost. This is the log of the reindex
2022-02-03 21:10:36,751 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is running
2022-02-03 21:10:37,182 [INFO] from org.janusgraph.graphdb.database.management.ManagementSystem in Thread-61 [|] Index update job successful for [global5]
2022-02-03 21:10:37,752 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is finished
2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-13 [|] Singleton manager starting singleton actor [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] Singleton manager starting singleton actor [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader]
2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] Singleton manager starting singleton actor [akka://application/user/flowSingletonManager/singleton]
2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-21 [|] ClusterSingletonManager state change [Start -> Oldest]
2022-02-03 21:10:38,619 [INFO] from play.api.Play in main [|] Application started (Prod) (no global state)
2022-02-03 21:10:38,942 [INFO] from play.core.server.AkkaHttpServer in main [|] Listening for HTTP on /0:0:0:0:0:0:0:0:8000
2022-02-03 21:10:38,943 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader]
2022-02-03 21:10:39,593 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,488 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton]
2022-02-03 21:10:51,636 [INFO] from org.thp.thehive.services.ObservableTypeIntegrityCheckOps in pool-12-thread-1 [|774df931] Found duplicate entities:
ObservableType(domain,false)
ObservableType(domain,false)
2022-02-03 21:11:32,575 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-13 [00000001|] 127.0.0.1 GET / took 14ms and returned 308 0 bytes
2022-02-03 21:11:32,648 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-4 [00000002|] 127.0.0.1 GET /index.html took 38ms and returned 200 1191 bytes
(.....)
2022-02-03 21:25:26,943 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|0956f1f9] Found duplicate entities:
Data(xxxxxxx,None)
Data(xxxxxxx,None)
2022-02-03 21:25:28,008 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|3eefbb8a] Found duplicate entities:
Data(xxxxxx,None)
Data(xxxxxx,None)
2022-02-03 21:25:28,505 [INFO] from org.thp.thehive.services.DataIntegrityCheckOps in pool-9-thread-1 [|04b74a3b] Found duplicate entities:
Steps to Reproduce
Configure Elasticsearch index
Force reindex with parameter 'db.janusgraph.forceDropAndRebuildIndex: true'
The observables that are considered duplicated disappear from the alert/case:
To-om
commented
2 years ago
Request Type
Bug
Work Environment
If I set the parameter 'db.janusgraph.forceDropAndRebuildIndex: true' when using Elasticsearch as index backend, after a while lots of entities are marked as duplicated and data from observables are lost. This is the log of the reindex
2022-02-03 21:10:36,751 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is running 2022-02-03 21:10:37,182 [INFO] from org.janusgraph.graphdb.database.management.ManagementSystem in Thread-61 [|] Index update job successful for [global5] 2022-02-03 21:10:37,752 [INFO] from org.thp.scalligraph.models.Database in application-akka.actor.default-dispatcher-13 [|mgmt-296117d3] Reindex job 178b1e59 is finished 2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-13 [|] Singleton manager starting singleton actor [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:37,891 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] ClusterSingletonManager state change [Start -> Oldest] 2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] Singleton manager starting singleton actor [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader] 2022-02-03 21:10:37,930 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-18 [|] ClusterSingletonManager state change [Start -> Oldest] 2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-4 [|] Singleton manager starting singleton actor [akka://application/user/flowSingletonManager/singleton] 2022-02-03 21:10:38,580 [INFO] from akka.cluster.singleton.ClusterSingletonManager in application-akka.actor.default-dispatcher-21 [|] ClusterSingletonManager state change [Start -> Oldest] 2022-02-03 21:10:38,619 [INFO] from play.api.Play in main [|] Application started (Prod) (no global state) 2022-02-03 21:10:38,942 [INFO] from play.core.server.AkkaHttpServer in main [|] Listening for HTTP on /0:0:0:0:0:0:0:0:8000 2022-02-03 21:10:38,943 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/system/singletonManagerCaseNumberLeader/CaseNumberLeader] 2022-02-03 21:10:39,593 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-21 [|] Singleton identified at [akka://application/user/flowSingletonManager/singleton] 2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,487 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,488 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,489 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,490 [INFO] from akka.cluster.singleton.ClusterSingletonProxy in application-akka.actor.default-dispatcher-4 [|] Singleton identified at [akka://application/user/integrityCheckSingletonManager/singleton] 2022-02-03 21:10:51,636 [INFO] from org.thp.thehive.services.ObservableTypeIntegrityCheckOps in pool-12-thread-1 [|774df931] Found duplicate entities:
Steps to Reproduce