search

TheHive-Project / Cortex

Cortex: a Powerful Observable Analysis and Active Response Engine
https://thehive-project.org
GNU Affero General Public License v3.0
1.28k stars 218 forks source link

Virustotal analyzer does not work on fresh install #401

Closed ssman783 closed 2 years ago

ssman783 commented 2 years ago

Request Type

Bug

Work Environment

Question Answer
OS version (server) Ubuntu 20
Cortex version / git hash 3.1.4-1
Package Type DEB

Problem Description

Installed Cortex following official documentation, and YT video. When running virustotal analyser (either Scan or GetReport), analysis fails with only error "Bad status: -1". API key implemented. Only other analyser i tried is AbuseIPDB which works fine.

Possible Solutions

Now i dont really remember, but i probably mixed guide steps with video, by history and what can i remember i executed sudo pip install -U pip setuptools && sudo pip3 install -U pip setuptools and then for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -r $I; done && for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -r $I || true; done

which did not work (but did "pip2 setuptools" and not "pip setuptools", therefore i mixed this up) and then executed only pip3 half of the command, which finished succesfully i thinl. Then i rerun whole "for" command, with pip2 and pip3 which finished OK, reqs already met. I installed some python related packages in between.

Is this the reason thet virustotal has issues to run? If so, should i remove/purge cortex and reinstall, or something else? I don't really know where to check for more info on how to fix it or what is causing fault (adding logs from /opt/cortex/application.log from time of executing analysis and also trace from when i rerun pip2 and pip3 requrements check).

Complementary information

executing for pip2 and pip3 reqs now:

Requirement already satisfied: datetime in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (4.3)
Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2.25.1)
Requirement already satisfied: thehive4py in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 4)) (1.8.0)
Requirement already satisfied: python-magic in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 5)) (0.4.22)
Requirement already satisfied: filetype in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 6)) (1.0.9)
Requirement already satisfied: zope.interface in /usr/local/lib/python2.7/dist-packages (from datetime->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (5.4.0)
Requirement already satisfied: pytz in /usr/local/lib/python2.7/dist-packages (from datetime->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (2021.1)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (4.0.0)Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (1.22)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2.10)
Requirement already satisfied: future in /usr/local/lib/python2.7/dist-packages (from thehive4py->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 4)) (0.18.2)
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from zope.interface->datetime->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (44.1.1)
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.Requirement already satisfied: cortexutils in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/responders/Wazuh/requirements.txt (line 1)) (2.1.0)
......
Requirement already satisfied: cortexutils in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 1)) (2.1.0)
Requirement already satisfied: future in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 2)) (0.18.2)
Requirement already satisfied: requests in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2.25.1)
Requirement already satisfied: virustotal-api in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 4)) (1.1.11)
Requirement already satisfied: python-magic in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 5)) (0.4.22)
Requirement already satisfied: filetype in /usr/local/lib/python2.7/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 6)) (1.0.9)
Requirement already satisfied: chardet<5,>=3.0.2 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (1.22)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2.10)
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support pip 21.0 will remove support for this functionality.
.....
Requirement already satisfied: cortexutils in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 1)) (2.1.0)
Requirement already satisfied: datetime in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (4.3)
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2.27.1)
Requirement already satisfied: thehive4py in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 4)) (1.8.1)
Requirement already satisfied: python-magic in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 5)) (0.4.22)
Requirement already satisfied: filetype in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 6)) (1.0.9)
Requirement already satisfied: zope.interface in /usr/lib/python3/dist-packages (from datetime->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (4.7.1)
Requirement already satisfied: pytz in /usr/local/lib/python3.8/dist-packages (from datetime->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 2)) (2021.3)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.8/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2.0.10)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (2019.11.28)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 3)) (1.26.8)
Requirement already satisfied: future in /usr/local/lib/python3.8/dist-packages (from thehive4py->-r Cortex-Analyzers/responders/VirustotalDownloader/requirements.txt (line 4)) (0.18.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
.......
Requirement already satisfied: cortexutils in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 1)) (2.1.0)
Requirement already satisfied: future in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 2)) (0.18.2)
Requirement already satisfied: requests in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2.26.0)
Requirement already satisfied: virustotal-api in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 4)) (1.1.11)
Requirement already satisfied: python-magic in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 5)) (0.4.22)
Requirement already satisfied: filetype in /usr/local/lib/python3.8/dist-packages (from -r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 6)) (1.0.9)
Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.8/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2.0.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.8/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (1.26.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests->-r Cortex-Analyzers/analyzers/VirusTotal/requirements.txt (line 3)) (2019.11.28)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

application.log:

2022-01-21 14:24:39,887 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 2 - request [POST http://172.29.124.96:9200/cortex_6/_search] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:39,987 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-57 - Looking for similar job in the last 10 minutes (worker=12a6ed3c998d2969a58d65494df802b7, dataType=url, data=Left(sme.sk), tlp=2, parameters={})
2022-01-21 14:24:40,632 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 2 - request [POST http://172.29.124.96:9200/cortex_6/_search] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,109 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 2 - request [POST http://172.29.124.96:9200/cortex_6/_doc?refresh=wait_for] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,311 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [POST http://172.29.124.96:9200/cortex_6/_search?scroll=60000ms] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,392 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [POST http://172.29.124.96:9200/_search/scroll] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,405 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [DELETE http://172.29.124.96:9200/_search/scroll/] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,460 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [POST http://172.29.124.96:9200/cortex_6/_search?scroll=60000ms] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,516 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [POST http://172.29.124.96:9200/_search/scroll] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,532 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [DELETE http://172.29.124.96:9200/_search/scroll/] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:41,791 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [POST http://172.29.124.96:9200/cortex_6/_search] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:42,154 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 2 - request [POST http://172.29.124.96:9200/cortex_6/_update/4b0GfX4BupObna2Db_5C?routing=4b0GfX4BupObna2Db_5C&refresh=wait_for&_source=true&retry_on_conflict=0] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:42,415 [INFO] from org.thp.cortex.services.ProcessJobRunnerSrv in application-akka.actor.default-dispatcher-56 - Execute /opt/cortex/Cortex-Analyzers/analyzers/VirusTotal/virustotal.py in /opt/cortex/Cortex-Analyzers/analyzers, timeout is 30 minutes
2022-01-21 14:24:42,448 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-61 - Job 4b0GfX4BupObna2Db_5C has be updated (JsDefined("InProgress"))
2022-01-21 14:24:44,216 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [POST http://172.29.124.96:9200/cortex_6/_search?scroll=60000ms] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:44,240 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 1 - request [DELETE http://172.29.124.96:9200/_search/scroll/] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:45,193 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 2 - request [POST http://172.29.124.96:9200/cortex_6/_update/4b0GfX4BupObna2Db_5C?routing=4b0GfX4BupObna2Db_5C&refresh=wait_for&_source=true&retry_on_conflict=5] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
2022-01-21 14:24:45,197 [INFO] from org.thp.cortex.services.AuditActor in application-akka.actor.default-dispatcher-61 - Job 4b0GfX4BupObna2Db_5C has be updated (JsDefined("Failure"))
2022-01-21 14:24:45,230 [INFO] from org.thp.cortex.services.JobSrv in application-akka.actor.default-dispatcher-56 - Job 4b0GfX4BupObna2Db_5C has finished with status Failure
2022-01-21 14:24:52,150 [WARN] from org.elasticsearch.client.RestClient in I/O dispatcher 2 - request [POST http://172.29.124.96:9200/cortex_6/_search] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security."]
ssman783 commented 2 years ago

Later on i got error regarding that Connection was reset by peer. Later found out it was blocked by company firewall. No issue with analyzer itself.