search

TheHive-Project / Cortex-Analyzers

Cortex Analyzers Repository
https://TheHive-Project.github.io/Cortex-Analyzers/
GNU Affero General Public License v3.0
433 stars 374 forks source link

[Bug] [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found #1088

Open grumo35 opened 2 years ago

grumo35 commented 2 years ago

Describe the bug When trying any analyzers this errors pops-up : image

/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py:1043: InsecureRequestWarning: Unverified HTTPS request is being made to host 'urlscan.io'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn(Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4262)During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 703, in urlopen httplib_response = self._make_request( File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 386, in _make_request self._validate_conn(conn) File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 1040, in _validate_conn conn.connect() File "/usr/local/lib/python3.9/dist-packages/urllib3/connection.py", line 416, in connect self.sock = ssl_wrap_socket( File "/usr/local/lib/python3.9/dist-packages/urllib3/util/ssl_.py", line 404, in ssl_wrap_socket raise SSLError(e)urllib3.exceptions.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4262)During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 440, in send resp = conn.urlopen( File "/usr/local/lib/python3.9/dist-packages/urllib3/connectionpool.py", line 785, in urlopen retries = retries.increment( File "/usr/local/lib/python3.9/dist-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='urlscan.io', port=443): Max retries exceeded with url: /api/v1/result/538c6a57-2167-4d77-921b-d7e54bc08203/ (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4262)')))During handling of the above exception, another exception occurred:Traceback (most recent call last): File "/etc/cortex/analyzers/Urlscan.io/urlscan_analyzer.py", line 102, in <module> UrlscanAnalyzer().run() File "/etc/cortex/analyzers/Urlscan.io/urlscan_analyzer.py", line 42, in run 'indicator': self.scan(query) File "/etc/cortex/analyzers/Urlscan.io/urlscan_analyzer.py", line 30, in scan res = Urlscan(indicator).scan(self.api_key) File "/etc/cortex/analyzers/Urlscan.io/urlscan.py", line 38, in scan submission_req = requests.get(submission_url) File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 75, in get return request('get', url, params=params, **kwargs) File "/usr/local/lib/python3.9/dist-packages/requests/api.py", line 61, in request return session.request(method=method, url=url, **kwargs) File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 529, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.9/dist-packages/requests/sessions.py", line 645, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.9/dist-packages/requests/adapters.py", line 517, in send raise SSLError(e, request=request)requests.exceptions.SSLError: HTTPSConnectionPool(host='urlscan.io', port=443): Max retries exceeded with url: /api/v1/result/538c6a57-2167-4d77-921b-d7e54bc08203/ (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4262)')))

To Reproduce Steps to reproduce the behavior:

  1. Install cortex via DEB package version 3.1.4-1
  2. Clone this repository, export and configure analyzers in application.conf
  3. After installing the requirements of some analyzers they all fail due to SSLError

Complementary information URLHaus : image AbuseIPDB : image

Which certificate is he failing to get ? How ? '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1123)'))

Output of requests from the same server :

>>> import requests
>>> requests.get("https://urlscan.io")
<Response [200]>

This is beyond me i have tried the fix https://github.com/TheHive-Project/Cortex-Analyzers/issues/605 nothing changes.

Work environment

Possible solutions

I have tried certifi adding each CA certs from my own store to a custom pem file and it's not working either. It looks like cortex cannot figure out to find a certificate to trust.

so-what commented 2 years ago

Possible solutions

I have tried certifi adding each CA certs from my own store to a custom pem file and it's not working either. It looks like cortex cannot figure out to find a certificate to trust.

Please advise me the detailed steps how to add CA certs. I'm seeing similar error message when using Investigate_Sample_1_0

Unexpected Error: HTTPSConnectionPool(host='investigate.api.umbrella.com', port=443): Max retries exceeded with url: /sample/ec393726fb9749558c2fd816c41f0fa0c17ca79ecf43085eba27f17cbccecebe (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4296)')))

Work environment

grumo35 commented 2 years ago

Possible solutions I have tried certifi adding each CA certs from my own store to a custom pem file and it's not working either. It looks like cortex cannot figure out to find a certificate to trust.

Please advise me the detailed steps how to add CA certs. I'm seeing similar error message when using Investigate_Sample_1_0

Unexpected Error: HTTPSConnectionPool(host='investigate.api.umbrella.com', port=443): Max retries exceeded with url: /sample/ec393726fb9749558c2fd816c41f0fa0c17ca79ecf43085eba27f17cbccecebe (Caused by SSLError(SSLError(136, '[X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4296)')))

Work environment

  • Ubuntu 20.04.4 LTS
  • Cortex 3.1.6-1 (Installed deb package, also tested 3.1.4-1)

Hey, i've worked on the case and found out that if you add the CA of each analyzers url e.g :

  1. Download the CA cert of umbrella.com
  2. Add the CA cert in the trusted CA of cortex when configuring the analyzer

Tell me if you're having trouble solving for this analyzer

grumo35 commented 2 years ago

It really looks like the certifi library is not used by cortex-analyzers request, this has to be the root cause

so-what commented 2 years ago

@grumo35 Thank you very much for the information. Would you please elaborate "Add the CA cert in the trusted CA of cortex when configuring the analyzer" ? Where can I find "trusted CA of cortex" location?

grumo35 commented 2 years ago

@grumo35 Thank you very much for the information. Would you please elaborate "Add the CA cert in the trusted CA of cortex when configuring the analyzer" ? Where can I find "trusted CA of cortex" location?

In MISP analyzers configuration you have the option to add custom CA certs on each analyzers