search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
380 stars 412 forks source link

External connectors not working behind a proxy - example cisa-known-exploited-vulnerabilities #2088

Closed martinlindstrand closed 5 months ago

martinlindstrand commented 6 months ago

Description

We are working behind proxy. We are running from docker compose setup.

Environment

  1. OS Red hat 8.9
  2. OpenCTI version: 6.0.10
  3. Other environment details:

Behind a proxy, using default setup from https://docs.opencti.io/latest/deployment/installation/ except adding adding external connectors. cisa-known-exploited-vulnerabilities in the docker-compose.yml and adding support for proxy. http_proxy=http://proxy.com < not the real proxy adress. https_proxy=http://proxy.com no_proxy = internalipadresses....

Steps to create the smallest reproducible scenario: Adding the external connectors cisa-known-exploited-vulnerabilities.

Expected Output

It should import the information from cisa

Actual Output

from Cisa: {"timestamp": "2024-04-27T04:32:51.027442Z", "level": "ERROR", "name": "CISA Known Exploited Vulnerabilities", "message": "the JSON object must be str, bytes or bytearray, not NoneType", "exc_info": "Traceback (most recent call last):\n File \"/opt/opencti-connector-cisa-known-exploited-vulnerabilities/main.py\", line 284, in process_data\n cisa_data = json.loads(cisa_data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/json/init.py\", line 339, in loads\n raise TypeError(f'the JSON object must be str, bytes or bytearray, '\nTypeError: the JSON object must be str, bytes or bytearray, not NoneType"} {"timestamp": "2024-04-27T04:35:59.434021Z", "level": "ERROR", "name": "CISA Known Exploited Vulnerabilities", "message": "Error retrieving url https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json: <urlopen error [Errno 110] Operation timed out>", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/urllib/request.py\", line 1348, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.11/http/client.py\", line 1303, in request\n

same issue on urlhous.

Additional information

If you adding in the main.py in cisa-known-exploited-vulnerabilities on line 80 below try:

     proxy_url = os.environ.get('http_proxy')
     proxies = {"http": proxy_url, "https": proxy_url}
     if proxy_url:
        response = requests.get(url, proxies=proxies)
        return response.text

don't forget to add import requests

It will start working. it will be using the proxy settings and fetch the information. I suggest this type of change should be check on all the external connectors to make it work with proxy.

Other external connectors like alienvault are working fine behind the proxy.

Screenshots (optional)

helene-nguyen commented 5 months ago

Hi @martinlindstrand, Thank you for your workaround. Maybe I can add another suggestion. Because environment variables are case sensitive in Python scripts, when you add them to your `docker-compose.yml' file, you need to make sure that any environment variable you add is uppercase.

For proxy, you need to add it as the following in you docker-compose.yml file:

- HTTP_PROXY=...
- HTTPS_PROXY=...
- NO_PROXY=...

Could you try by setting this in uppercase and give us an update?

martinlindstrand commented 5 months ago

That did not work.

For reference when using HTTP_PROXY HTTPS_PROXY NO_PROXY

The response for using this code: proxy_url = os.environ.get('http_proxy') proxies = {"http": proxy_url, "https": proxy_url} print(proxy_url)

RESPONSE: python main.py None <------ response from the print(proxy_url) python3 main.py None <------ response from the print(proxy_url)

When using with lowercase http_proxy https_proxy no_proxy

python main.py http://proxy.com <------ response from the print(proxy_url)

helene-nguyen commented 5 months ago

@martinlindstrand Following our discussion on Slack, I close this issue but it can be re-opened if needed :)

CharlesMagne2001 commented 5 months ago

Hello, I have the same problem. I am working under proxy. I tried to use the connectors : ipinfo and shodan. I here is my yml part:

opencti: image: opencti/plateform:6.0.5 ... ... connector-shodan: image: opencti/connector-shodan:6.1.10 environment:

and I get the same error below when I enrich a IPv4 from shodan or ipinfo : SHODAN : {"timestamp": "2024-06-11T11:23:17.739011Z", "level": "ERROR", "name": "Shodan", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py\", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py\", line 750, in read\n result = self.opencti.query(query, {\"id\": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type \"TrackingNumber\".', 'error_message': 'Unknown type \"TrackingNumber\".'}"}

IPINFO: {"timestamp": "2024-06-11T11:22:31.125056Z", "level": "ERROR", "name": "IpInfo", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py\", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py\", line 750, in read\n result = self.opencti.query(query, {\"id\": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type \"TrackingNumber\".', 'error_message': 'Unknown type \"TrackingNumber\".'}"} {"timestamp": "2024-06-11T11:23:17.758439Z", "level": "ERROR", "name": "IpInfo", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py\", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py\", line 750, in read\n result = self.opencti.query(query, {\"id\": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type \"TrackingNumber\".', 'error_message': 'Unknown type \"TrackingNumber\".'}"

Could you tell me if the issues are linked to proxies? How to fix them?

KanakSasak commented 3 months ago

same here with hybrid analysis :

{"timestamp": "2024-07-16T05:12:28.691124Z", "level": "ERROR", "name": "Hybrid Analysis (Sandbox Windows 10 64bit)", "message": "Error in message processing, reporting error to API", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/connector/opencti_connector_helper.py\", line 268, in _data_handler\n opencti_entity = do_read(id=entity_id, withFiles=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/entities/opencti_stix_cyber_observable.py\", line 158, in read\n result = self.opencti.query(query, {\"id\": id})\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 363, in query\n raise ValueError(value_error)\nValueError: {'name': 'Unknown type \"TrackingNumber\".', 'error_message': 'Unknown type \"TrackingNumber\".'}"}