search

TheHive-Project / TheHive

TheHive: a Scalable, Open Source and Free Security Incident Response Platform
https://thehive-project.org
GNU Affero General Public License v3.0
3.4k stars 618 forks source link

[Bug] The Hive4 <> Cortex Authentication Issue (INTERMITTENT) #1924

Open cyamal1b4 opened 3 years ago

cyamal1b4 commented 3 years ago

Bug report for The Hive4

System: OS Version Ubuntu 20.04

The Hive 4.1.2 Cortex 3.1.1

DEB

Chrome / Firefox / Edge

Problem: The Hive4 is properly configured to use several +50 analyzers with a local cortex server. Same OS (16+ GB RAM, 8 Cores). 75% of the time the analyzers work as expected, however, after some time throughout the middle of the workday if I load in a new domain in a case I was working on in the morning and run all my domain analyzers it takes a few minutes and eventually crashes the cortex connection (not the cortex server itself) the analyzers eventually finish on the Cortex Server but the results dont get sent to the Hive. Below is the log output from the cortex "application.log" during the time this is occuring. It seems to be complaining about authentication. This is an intermittent issue, it happens randomly throughout the day. Any thoughts?

Log:

2021-04-01 19:16:49,705 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-9052 - GET /api/job/xOvXjngBkXre5T9Ui9gd/waitreport?atMost=1%20second returned 401
org.elastic4play.AuthenticationError: Authentication failure
    at org.elastic4play.controllers.Authenticated.$anonfun$getContext$4(Authenticated.scala:272)
    at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-04-01 19:16:49,705 [INFO] from org.thp.cortex.services.ErrorHandler in application-akka.actor.default-dispatcher-9053 - GET /api/job/yuvXjngBkXre5T9Uj9gI/waitreport?atMost=1%20second returned 401
org.elastic4play.AuthenticationError: Authentication failure
    at org.elastic4play.controllers.Authenticated.$anonfun$getContext$4(Authenticated.scala:272)
    at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

Later

2021-04-01 19:37:00,936 [WARN] from org.elastic4play.database.SearchWithScroll in application-akka.actor.default-dispatcher-22 - Search error
org.elastic4play.SearchError: all shards failed
    at org.elastic4play.database.DBConfiguration.$anonfun$execute$2(DBConfiguration.scala:157)
    at scala.concurrent.Future.$anonfun$flatMap$1(Future.scala:307)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-04-01 19:37:00,936 [ERROR] from org.elastic4play.services.auth.MultiAuthSrv in application-akka.actor.default-dispatcher-8 - Authentication failure
org.elastic4play.AuthenticationError: Authentication using API key is not supported
    at org.elastic4play.services.AuthSrv.authenticate(UserSrv.scala:48)
    at org.elastic4play.services.AuthSrv.authenticate$(UserSrv.scala:47)
    at org.thp.cortex.services.LocalAuthSrv.authenticate(LocalAuthSrv.scala:19)
    at org.elastic4play.services.auth.MultiAuthSrv.$anonfun$authenticate$3(MultiAuthSrv.scala:58)
    at org.elastic4play.services.auth.MultiAuthSrv$$anonfun$$nestedInanonfun$forAllAuthProvider$1$1.applyOrElse(MultiAuthSrv.scala:43)
    at org.elastic4play.services.auth.MultiAuthSrv$$anonfun$$nestedInanonfun$forAllAuthProvider$1$1.applyOrElse(MultiAuthSrv.scala:41)
    at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)

=> ElasticError(404,404,None,None,None,List(),None,None,None,List())
2021-04-01 19:38:53,473 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-22 - Register new listener for job 3-vujngBkXre5T9UxPml (Actor[akka://application/temp/audit$p])
2021-04-01 19:38:54,483 [ERROR] from org.elastic4play.database.DBConfiguration in application-akka.actor.default-dispatcher-11 - ElasticSearch request failure: DELETE:/_search/scroll/?
StringEntity({"scroll_id":["FGluY2x1ZGVfY29udGV4dF91dWlkDnF1ZXJ5VGhlbkZldGNoBRYySEFzWV91b1NVYV9CYzlXWi1HaWpnAAAAAAABIV8WbGN2eE1YT1ZRbzZNN2xTcnU0Z280URYySEFzWV91b1NVYV9CYzlXWi1HaWpnAAAAAAABIWEWbGN2eE1YT1ZRbzZNN2xTcnU0Z280URYySEFzWV91b1NVYV9CYzlXWi1HaWpnAAAAAAABIWIWbGN2eE1YT1ZRbzZNN2xTcnU0Z280URYySEFzWV91b1NVYV9CYzlXWi1HaWpnAAAAAAABIWMWbGN2eE1YT1ZRbzZNN2xTcnU0Z280URYySEFzWV91b1NVYV9CYzlXWi1HaWpnAAAAAAABIWAWbGN2eE1YT1ZRbzZNN2xTcnU0Z280UQ=="]},Some(application/json))

Later++

2021-04-01 19:41:37,683 [ERROR] from org.elastic4play.services.auth.MultiAuthSrv in application-akka.actor.default-dispatcher-21 - Authentication failure
org.elastic4play.AuthenticationError: Authentication using API key is not supported
    at org.elastic4play.services.AuthSrv.authenticate(UserSrv.scala:48)
    at org.elastic4play.services.AuthSrv.authenticate$(UserSrv.scala:47)
    at org.thp.cortex.services.LocalAuthSrv.authenticate(LocalAuthSrv.scala:19)
    at org.elastic4play.services.auth.MultiAuthSrv.$anonfun$authenticate$3(MultiAuthSrv.scala:58)
    at org.elastic4play.services.auth.MultiAuthSrv$$anonfun$$nestedInanonfun$forAllAuthProvider$1$1.applyOrElse(MultiAuthSrv.scala:43)
    at org.elastic4play.services.auth.MultiAuthSrv$$anonfun$$nestedInanonfun$forAllAuthProvider$1$1.applyOrElse(MultiAuthSrv.scala:41)
    at scala.concurrent.Future.$anonfun$recoverWith$1(Future.scala:417)
    at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
    at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:56)
    at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:93)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
    at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:93)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:48)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
2021-04-01 19:41:38,389 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Job 8evujngBkXre5T9U0PnE has be updated (JsDefined("Success"))
2021-04-01 19:41:38,389 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Job 2evujngBkXre5T9UwPkn has be updated (JsDefined("Success"))
2021-04-01 19:41:38,390 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-18 - Job 8evujngBkXre5T9U0PnE has finished with status Success
2021-04-01 19:41:38,390 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-8 - Job 2evujngBkXre5T9UwPkn has finished with status Success
2021-04-01 19:41:38,396 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Job 3OvujngBkXre5T9UwPk_ has be updated (JsDefined("Success"))
2021-04-01 19:41:38,397 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-18 - Job 3OvujngBkXre5T9UwPk_ has finished with status Success
2021-04-01 19:41:38,402 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-10 - Job 1-vujngBkXre5T9UwPki has finished with status Success
2021-04-01 19:41:38,402 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Job 6OvujngBkXre5T9UyPm6 has be updated (JsDefined("Success"))
2021-04-01 19:41:38,402 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Job 1-vujngBkXre5T9UwPki has be updated (JsDefined("Success"))
2021-04-01 19:41:38,402 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-18 - Job 6OvujngBkXre5T9UyPm6 has finished with status Success
2021-04-01 19:41:38,410 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-11 - Job 9OvujngBkXre5T9U0Pnz has finished with status Success
2021-04-01 19:41:38,410 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-11 - Job 9OvujngBkXre5T9U0Pnz has be updated (JsDefined("Success"))
2021-04-01 19:41:38,416 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Job 4-vujngBkXre5T9Ux_nv has be updated (JsDefined("Success"))
2021-04-01 19:41:38,416 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-10 - Job 4-vujngBkXre5T9Ux_nv has finished with status Success
2021-04-01 19:41:38,630 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-22 - Unregister listener for job 9OvujngBkXre5T9U0Pnz (Actor[akka://application/temp/audit$P])
2021-04-01 19:41:38,640 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-22 - Unregister listener for job 7-vujngBkXre5T9U0Pmz (Actor[akka://application/temp/audit$Q])
2021-04-01 19:41:38,640 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 8-vujngBkXre5T9U0PnQ (Actor[akka://application/temp/audit$S])
2021-04-01 19:41:38,640 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 7OvujngBkXre5T9Uz_n4 (Actor[akka://application/temp/audit$T])
2021-04-01 19:41:38,641 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 5OvujngBkXre5T9Ux_n3 (Actor[akka://application/temp/audit$U])
2021-04-01 19:41:38,641 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 8uvujngBkXre5T9U0PnF (Actor[akka://application/temp/audit$V])
2021-04-01 19:41:38,641 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 9evujngBkXre5T9U1PkG (Actor[akka://application/temp/audit$W])
2021-04-01 19:41:38,641 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-12 - Unregister listener for job 8OvujngBkXre5T9U0Pm9 (Actor[akka://application/temp/audit$R])
2021-04-01 19:41:38,649 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 5evujngBkXre5T9UyPkf (Actor[akka://application/temp/audit$X])
2021-04-01 19:41:38,649 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 8evujngBkXre5T9U0PnE (Actor[akka://application/temp/audit$Y])
2021-04-01 19:41:38,649 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 2evujngBkXre5T9UwPkn (Actor[akka://application/temp/audit$0])
2021-04-01 19:41:38,649 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 7uvujngBkXre5T9U0PlD (Actor[akka://application/temp/audit$Z])
2021-04-01 19:41:38,649 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-10 - Unregister listener for job 6OvujngBkXre5T9UyPm6 (Actor[akka://application/temp/audit$1])
2021-04-01 19:41:38,660 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Unregister listener for job 3OvujngBkXre5T9UwPk_ (Actor[akka://application/temp/audit$2])
2021-04-01 19:41:38,660 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Unregister listener for job 1-vujngBkXre5T9UwPki (Actor[akka://application/temp/audit$4])
2021-04-01 19:41:38,660 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-9 - Unregister listener for job 4-vujngBkXre5T9Ux_nv (Actor[akka://application/temp/audit$3])
2021-04-01 19:41:40,449 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-22 - Job 7-vujngBkXre5T9U0Pmz has be updated (JsDefined("Success"))
2021-04-01 19:41:40,449 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-10 - Job 7-vujngBkXre5T9U0Pmz has finished with status Success
nadouani commented 3 years ago 
2021-04-01 19:41:37,683 [ERROR] from org.elastic4play.services.auth.MultiAuthSrv in application-akka.actor.default-dispatcher-21 - Authentication failure
org.elastic4play.AuthenticationError: Authentication using API key is not supported

This looks weird.

cyamal1b4 commented 3 years ago

It's a strange issue. I've used The Hive / Cortex for some time now, just recently migrated up to 4.X.X. What is most strange to me is that it is intermittent. I could run it in an hour against all of them and it could work. Seems to be domains more than IPs as well that it gets upset about... very strange

cyamal1b4 commented 3 years ago

On the Cortex instance, they finished successfully as well.

cyamal1b4 commented 3 years ago

Addtionally, a reboot of the system brought the analysis into the hive. I had results for the domain I queried once I restarted. Very Strange.

Also I have noticed The Hive not rendering long reports when I click on the short report tags. This is the error for that:

For instance I selected the PT:Malware tag (short report) and it didnt present the long report. I selected the Cyberprotect short report and it presented

2021-04-01 20:32:02,143 [WARN] from org.thp.scalligraph.query.InputFilter in application-akka.actor.default-dispatcher-18 [0000007f|] Use of filter {"analyzerId": "FString(Cyberprotect_ThreatScore_1_0)"} is deprecated. Please use {"_field": "analyzerId", "_value": "FString(Cyberprotect_ThreatScore_1_0)"}
2021-04-01 20:32:02,178 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [0000007f|] 10.3.1.1 POST /api/v1/query?name=observable-jobs-~40964160 took 42ms and returned 200
2021-04-01 20:32:09,960 [WARN] from org.thp.scalligraph.query.InputFilter in application-akka.actor.default-dispatcher-18 [00000080|] Use of filter {"analyzerId": "FString(Cyberprotect_ThreatScore_1_0)"} is deprecated. Please use {"_field": "analyzerId", "_value": "FString(Cyberprotect_ThreatScore_1_0)"}
2021-04-01 20:32:09,966 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [00000080|] 10.3.1.1 POST /api/v1/query?name=observable-jobs-~40964160 took 13ms and returned 200
2021-04-01 20:32:13,867 [WARN] from org.thp.scalligraph.query.InputFilter in application-akka.actor.default-dispatcher-18 [00000081|] Use of filter {"analyzerId": "FString(PassiveTotal_Malware_2_0)"} is deprecated. Please use {"_field": "analyzerId", "_value": "FString(PassiveTotal_Malware_2_0)"}
2021-04-01 20:32:13,901 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [00000081|] 10.3.1.1 POST /api/v1/query?name=observable-jobs-~40964160 took 38ms and returned 200
2021-04-01 20:32:13,933 [WARN] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-18 [|] An error occurs (Neither the sideEffects, map, nor path has a 81487033-1670-4af1-85e1-298b7c85093c-key: WherePredicateStep(eq(81487033-1670-4af1-85e1-298b7c85093c))), retrying (1)
nadouani commented 3 years ago

Ok we will investigate

cyamal1b4 commented 3 years ago

@nadouani @To-om I know you and team are diligently working this effort. In being a good steward of your time and potential past issues, I found this issue: https://github.com/TheHive-Project/Cortex/issues/259

I had recenty added in all of the passive total analyzers, some 8-10 in total. Some werent working and I was in the process of taking a look to see which. That said if its an analyzer issue that may help scope your t-shooting. I'll get back with you guys and see what happens if I turn the passive total ones all off (im not a paying customer anyway, was just using the free tier and they may not like that.. lol )

nadouani commented 3 years ago

@cyamal1b4 thanks for pointing this. Do you have your own analyzers or the potentially malformed one is part of the existing list?

cyamal1b4 commented 3 years ago

@nadouani It was the full gambit of PassiveTotal Analyzers. But we just did a fresh test and re-ran another IP against all the IP analysers (~25 or so) and it failed out the same way. So that hypothesis may have not been right on my end, the faulty analyzer bit.

danniranderis commented 3 years ago

We have experienced errors similar to this, using THP 4.0.5, Cortex 3.1.1 and OS Ubuntu 20.04 (THP and Cortex on two different servers, no clustering).

I especially see this type of error, where THP's connection to Cortex is reported as down when we start multiple analyzers (like 15+ ish). From time to time we have analyzers failing, but it doesn't seem to trigger this behaviour. Many analyzers in a row does however.

cyamal1b4 commented 3 years ago

I've also noticed it seems to know if i've ran the analyzers against an observable already and has been prone to errors there as well with this recent update. Maybe the indexing from the hive talking with cortex jobs with elastic is causing errors if it thinks there are duplicates. Just brainstorming.

nadouani commented 3 years ago

We need to get this out of 4.1.3 as it needs to be investigated and we don't want to block the release.

cyamal1b4 commented 3 years ago

@nadouani I certainly understand that. The larger issue for me now seems to be this one from above, as your team progresses I just wanted to highlight this one. It does appear to be random and it wipes the record that the analysis happened from the Hive as well. Forcing it to be ran again.

Also I have noticed The Hive not rendering long reports when I click on the short report tags. This is the error for that:

For instance I selected the PT:Malware tag (short report) and it didnt present the long report. I selected the Cyberprotect short report and it presented

2021-04-01 20:32:02,143 [WARN] from org.thp.scalligraph.query.InputFilter in application-akka.actor.default-dispatcher-18 [0000007f|] Use of filter {"analyzerId": "FString(Cyberprotect_ThreatScore_1_0)"} is deprecated. Please use {"_field": "analyzerId", "_value": "FString(Cyberprotect_ThreatScore_1_0)"}
2021-04-01 20:32:02,178 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [0000007f|] 10.3.1.1 POST /api/v1/query?name=observable-jobs-~40964160 took 42ms and returned 200
2021-04-01 20:32:09,960 [WARN] from org.thp.scalligraph.query.InputFilter in application-akka.actor.default-dispatcher-18 [00000080|] Use of filter {"analyzerId": "FString(Cyberprotect_ThreatScore_1_0)"} is deprecated. Please use {"_field": "analyzerId", "_value": "FString(Cyberprotect_ThreatScore_1_0)"}
2021-04-01 20:32:09,966 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [00000080|] 10.3.1.1 POST /api/v1/query?name=observable-jobs-~40964160 took 13ms and returned 200
2021-04-01 20:32:13,867 [WARN] from org.thp.scalligraph.query.InputFilter in application-akka.actor.default-dispatcher-18 [00000081|] Use of filter {"analyzerId": "FString(PassiveTotal_Malware_2_0)"} is deprecated. Please use {"_field": "analyzerId", "_value": "FString(PassiveTotal_Malware_2_0)"}
2021-04-01 20:32:13,901 [INFO] from org.thp.scalligraph.AccessLogFilter in application-akka.actor.default-dispatcher-18 [00000081|] 10.3.1.1 POST /api/v1/query?name=observable-jobs-~40964160 took 38ms and returned 200
2021-04-01 20:32:13,933 [WARN] from org.thp.scalligraph.utils.Retry in application-akka.actor.default-dispatcher-18 [|] An error occurs (Neither the sideEffects, map, nor path has a 81487033-1670-4af1-85e1-298b7c85093c-key: WherePredicateStep(eq(81487033-1670-4af1-85e1-298b7c85093c))), retrying (1)
diogodaz commented 2 years ago

@nadouani any news for hive 4.2.0 of this fix?

Version: Docker debian 10 Hive 4.1.11-1 Cortex 3.1.1 Cassandra 3.11.11

Problem: 2021-10-29 16:55:20,702 [ERROR] from org.elastic4play.services.auth.MultiAuthSrv in application-akka.actor.default-dispatcher-8397 - Authentication failur org.elastic4play.AuthenticationError: Authentication using API key is not supported