Closed padey closed 8 months ago
2024-01-02 08:44:36,495 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in application-analyzer-20 - Execute /opt/Cortex-Analyzers/analyzers/VirusTotal/virustotal.py in /opt/Cortex-Analyzers/analyzers, timeout is 30 minutes
2024-01-02 08:44:37,901 [INFO] from org.thp.cortex.services.AccessLogFilter in application-akka.actor.default-dispatcher-6 - 192.168.XXX.XXX POST /api/job/status took 13ms and returned 200 325 bytes
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: Traceback (most recent call last):
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/opt/Cortex-Analyzers/analyzers/VirusTotal/virustotal.py", line 407, in <module>
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: VirusTotalAnalyzer().run()
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/opt/Cortex-Analyzers/analyzers/VirusTotal/virustotal.py", line 365, in run
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: self.report(results)
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/local/lib/python3.10/dist-packages/cortexutils/analyzer.py", line 110, in report
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: super(Analyzer, self).report({
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/local/lib/python3.10/dist-packages/cortexutils/worker.py", line 203, in report
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: self.__write_output(output, ensure_ascii=ensure_ascii)
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/local/lib/python3.10/dist-packages/cortexutils/worker.py", line 127, in __write_output
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: json.dump(data, f_output, ensure_ascii=ensure_ascii)
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/__init__.py", line 179, in dump
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: for chunk in iterable:
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/encoder.py", line 431, in _iterencode
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: yield from _iterencode_dict(o, _current_indent_level)
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
2024-01-02 08:44:40,515 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: yield from chunks
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: yield from chunks
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/encoder.py", line 405, in _iterencode_dict
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: yield from chunks
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/encoder.py", line 438, in _iterencode
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: o = _default(o)
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: File "/usr/lib/python3.10/json/encoder.py", line 179, in default
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: raise TypeError(f'Object of type {o.__class__.__name__} '
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: TypeError: Object of type WhistleBlowerDict is not JSON serializable
2024-01-02 08:44:40,525 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: Unclosed client session
2024-01-02 08:44:40,525 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: client_session: <aiohttp.client.ClientSession object at 0x7f17c02b47c0>
2024-01-02 08:44:40,525 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: Unclosed connector
2024-01-02 08:44:40,526 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f17c0296740>, 337096.023685094)]']
2024-01-02 08:44:40,526 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: connector: <aiohttp.connector.TCPConnector object at 0x7f17c1ffb7f0>
2024-01-02 08:44:40,555 [ERROR] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-4 - Job uidYyYwBF66zXQbdliQO has failed
com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input within/between Object entries
at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 1917]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:682)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon2(UTF8StreamJsonParser.java:3202)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon(UTF8StreamJsonParser.java:3117)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:802)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:229)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:143)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:138)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2831)
at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:288)
at play.api.libs.json.StaticBinding$.parseJsValue(StaticBinding.scala:21)
at play.api.libs.json.Json$.parse(Json.scala:175)
at org.thp.cortex.services.JobRunnerSrv.extractReport(JobRunnerSrv.scala:163)
at org.thp.cortex.services.JobRunnerSrv.$anonfun$run$13(JobRunnerSrv.scala:247)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177)
This is the log from the Python. I guess there are some errors round about the report rendering / processing?
2024-01-02 08:44:40,516 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: TypeError: Object of type WhistleBlowerDict is not JSON serializable
2024-01-02 08:44:40,525 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: Unclosed client session
2024-01-02 08:44:40,525 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: client_session: <aiohttp.client.ClientSession object at 0x7f17c02b47c0>
2024-01-02 08:44:40,525 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: Unclosed connector
2024-01-02 08:44:40,526 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f17c0296740>, 337096.023685094)]']
2024-01-02 08:44:40,526 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in Thread-26 - Job uidYyYwBF66zXQbdliQO: connector: <aiohttp.connector.TCPConnector object at 0x7f17c1ffb7f0>
Just installed a complete new Cortex Instance. Same error:
2024-01-02 10:08:12,539 [ERROR] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-6 - Job H_qkyYwBYLRkmdOhlOBV has failed
com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input within/between Object entries
at [Source: (sun.nio.ch.ChannelInputStream); line: 1, column: 3817]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:682)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon2(UTF8StreamJsonParser.java:3202)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon(UTF8StreamJsonParser.java:3117)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:802)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:229)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:143)
at play.api.libs.json.jackson.JsValueDeserializer.deserialize(JacksonJson.scala:138)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:323)
at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:4650)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2831)
at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:288)
at play.api.libs.json.StaticBinding$.parseJsValue(StaticBinding.scala:21)
at play.api.libs.json.Json$.parse(Json.scala:175)
at org.thp.cortex.services.JobRunnerSrv.extractReport(JobRunnerSrv.scala:163)
at org.thp.cortex.services.JobRunnerSrv.$anonfun$run$13(JobRunnerSrv.scala:247)
at scala.concurrent.impl.Promise.$anonfun$transformWith$1(Promise.scala:41)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:63)
at akka.dispatch.BatchingExecutor$BlockableBatch.$anonfun$run$1(BatchingExecutor.scala:100)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:85)
at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:100)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Hello Same error here, i notice when was realised the new docker image 3, instead of 3.0 https://hub.docker.com/r/cortexneurons/virustotal_getreport/tags
Any solution?
ping: @nadouani @To-om
Hi have solved by using the previous docker image 3.0, by custom analyzer json configuration with docker image.
Hi @igorg1312 we will investigate this.
Hello there,
The issue comes from an update of the VirusTotal python library (vt-py). The analyzer currently works fine with vt-py 0.17.5. The bug comes with 0.18.0. We are preparing a fix. Thanks for reporting this issue !
Regards
Hello @cyberpescadito thanks for the quick response. We noticed also that the score in the tags from the last_analysis_stats isn't available anymore also with docker image 3.0, is this also related to vt-py 0.18.0 as well?
@cyberpescadito - perfect, thanks! :)
Hello @cyberpescadito thanks for the quick response. We noticed also that the score in the tags from the last_analysis_stats isn't available anymore also with docker image 3.0, is this also related to vt-py 0.18.0 as well?
Hello @igorg1312 , I see this last_analysis_stats in my full report using analyzer GetReport (3.1):
Is it what you are looking for?
Let me know if we're not talking about the same propertu
@cyberpescadito any updates on the fix? :)
@cyberpescadito any updates on the fix? :)
A fix has just been released by @jeromeleonard :)
@cyberpescadito @jeromeleonard looking good!
thx for the feedback.
Request Type
Bug
Work Environment
Problem Description
Since 28.12.23 the Virustotal Get Report / Scan Analyzer is broken. Only if VirusTotal returns "not found", the job finishes successfully, otherwise it goes to status "failed" but in Cortex web interface it remains on "In Progress".
Steps to Reproduce
Complementary information
The Log from Cortex shows the following: