search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
374 stars 407 forks source link

[Data not displayed] Error in custom connector #2398

Closed meetghodasara closed 2 months ago

meetghodasara commented 3 months ago

Description

I have create the custom_connector with the docker configuration given below.

Docker configuration.

version: '3'
services:
  connector:
    build: .
    container_name: connector
    environment:
      # Connector's definition parameters:
      - CONNECTOR_NAME=custom_connector
      - CONNECTOR_SCOPE=stix2
      # Connector's generic execution parameters:
      - OPENCTI_URL=http://localhost
      - OPENCTI_TOKEN=e801b101-ef00-4e24-9593-1d32911bace9
      - CONNECTOR_ID=2f3558fc-6eb7-413f-9ae0-5cab8a38cbab
      - CONNECTOR_CONFIDENCE_LEVEL=100 # From 0 (Unknown) to 100 (Fully trusted).
      - CONNECTOR_LOG_LEVEL=info
      - CONNECTOR_RUN_EVERY=60s
      # Connector's custom execution parameters:
      - EXTRA_PARAMETER=foobar
    restart: always

networks:
  default:
    external: true
    name: docker_default

main.py

# import os
from datetime import datetime
import os
import sys
import time

import stix2
import yaml
from src.lib.external_import import ExternalImportConnector
from pycti import OpenCTIConnectorHelper, get_config_variable

class CustomConnector(ExternalImportConnector):
    def __init__(self):
                # Instantiate the connector helper from config
        config_file_path = os.path.dirname(os.path.abspath(__file__)) + "/config.yml"
        config = (
            yaml.load(open(config_file_path), Loader=yaml.SafeLoader)
            if os.path.isfile(config_file_path)
            else {}
        )
        self.helper = OpenCTIConnectorHelper(config)
        self.interval = '10s'

    def _collect_intelligence(self) -> []:
        """Collects intelligence from channels
        Add your code depending on the use case as stated at https://docs.opencti.io/latest/development/connectors/.
        Some sample code is provided as a guide to add a specific observable and a reference to the main object.
        Consider adding additional methods to the class to make the code more readable.
        Returns:
            stix_objects: A list of STIX2 objects."""
        self.helper.log_debug(
            f"{self.helper.connect_name} connector is starting the collection of objects..."
        )
        stix_objects = []

        # ===========================
        # === Add your code below ===
        # ===========================
        self.helper.log_debug("Creating a sample reference using STIX2...")
        main_reference = stix2.ExternalReference(
            source_name="GitHub",
            url="https://github.com/OpenCTI-Platform/connectors",
            description="A sample external reference used by the connector.",
        )

        self.helper.log_debug("Creating an observable for the IPv4...")
        ipv4_observable = stix2.IPv4Address(
            value="2.2.2.2",
            object_marking_refs=[stix2.TLP_GREEN],
            custom_properties={
                "description": "A sample observable created for the tutorial.",
                "labels": ["test", "tutorial"],
                "x_opencti_create_indicator": False,
                "external_references": [main_reference],
            },
        )
        stix_objects.append(ipv4_observable)
        bundle = self.helper.stix2_create_bundle(stix_objects)
        # ===========================
        # === Add your code above ===
        # ===========================
        timestamp = int(time.time())
        self.helper.log_info(
            f"{len(stix_objects)} STIX2 objects have been compiled by {self.helper.connect_name} connector. "
        )
        now = datetime.fromtimestamp(timestamp)
        friendly_name = "Custom connector run @ " + now.strftime(
                        "%Y-%m-%d %H:%M:%S"
                    )
        work_id = self.helper.api.work.initiate_work(
                        self.helper.connect_id, friendly_name
                    )
        self.helper.send_stix2_bundle(
                            bundle,work_id=work_id,
                        )
        message = "Connector successfully run, storing last_run as " + str(
                        timestamp
                    )
        self.helper.api.work.to_processed(work_id, message)
        return stix_objects

    def get_interval(self) -> int:
        return int(self.interval) * 60 * 60 * 24

if __name__ == "__main__":
    try:
        connector = CustomConnector()
        connector.run()
    except Exception as e:
        print(e)
        time.sleep(10)
        sys.exit(0)

Environment

  1. OS : Ubuntu 22.04.4 LTS
  2. OpenCTI version: 6.2.7
  3. OpenCTI client: python

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Complete setup using docker in Linux VM. Setup documentation
  2. After running the custom connector with the python. I get an this issue. I have provided all groups and user to all TLP.
  3. Error 
    {"timestamp": "2024-07-24T18:51:21.370549Z", "level": "ERROR", "name": "pika.adapters.utils.selector_ioloop_adapter", "message": "Address resolution failed: gaierror(-3, 'Temporary failure in name resolution')"}
{"timestamp": "2024-07-24T18:51:21.370944Z", "level": "ERROR", "name": "pika.adapters.utils.connection_workflow", "message": "getaddrinfo failed: gaierror(-3, 'Temporary failure in name resolution')."}
{"timestamp": "2024-07-24T18:51:21.371078Z", "level": "ERROR", "name": "pika.adapters.utils.connection_workflow", "message": "AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-3, 'Temporary failure in name resolution'); first exception - None."}
{"timestamp": "2024-07-24T18:51:21.371151Z", "level": "ERROR", "name": "pika.adapters.utils.connection_workflow", "message": "AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-3, 'Temporary failure in name resolution'); first exception - None"}
{"timestamp": "2024-07-24T18:51:21.371237Z", "level": "ERROR", "name": "pika.adapters.blocking_connection", "message": "Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-3, 'Temporary failure in name resolution'); first exception - None"}
{"timestamp": "2024-07-24T18:51:21.371435Z", "level": "ERROR", "name": "pika.adapters.blocking_connection", "message": "Error in _create_connection().", "exc_info": "Traceback (most recent call last):\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py\", line 451, in _create_connection\n    raise self._reap_last_connection_workflow_error(error)\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/utils/selector_ioloop_adapter.py\", line 565, in _resolve\n    result = socket.getaddrinfo(self._host, self._port, self._family,\n  File \"/usr/lib/python3.10/socket.py\", line 955, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -3] Temporary failure in name resolution"}
{"timestamp": "2024-07-24T18:51:21.371753Z", "level": "ERROR", "name": "Common Vulnerabilities and Exposures", "message": "[Errno -3] Temporary failure in name resolution", "exc_info": "Traceback (most recent call last):\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/src/lib/external_import.py\", line 132, in run\n    bundle_objects = self._collect_intelligence()\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/main.py\", line 80, in _collect_intelligence\n    self.helper.send_stix2_bundle(\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pycti/connector/opencti_connector_helper.py\", line 1306, in send_stix2_bundle\n    pika_connection = pika.BlockingConnection(pika_parameters)\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py\", line 360, in __init__\n    self._impl = self._create_connection(parameters, _impl_class)\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py\", line 451, in _create_connection\n    raise self._reap_last_connection_workflow_error(error)\n  File \"/home/devuser/openCTI/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/utils/selector_ioloop_adapter.py\", line 565, in _resolve\n    result = socket.getaddrinfo(self._host, self._port, self._family,\n  File \"/usr/lib/python3.10/socket.py\", line 955, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -3] Temporary failure in name resolution"}

Expected Output

The data should be inserted and displayed in the Observable objects. The dashboard should have observable objects.

Actual Output

  1. Ingestion - connectors shows the activity log but it always show in-progress.

    Screenshot 2024-07-25 at 9 00 49 AM

  2. The Observable objects are empty here.

    Screenshot 2024-07-25 at 9 02 53 AM

  3. The activity log contains these activity.

    Screenshot 2024-07-25 at 9 03 40 AM

Additional information

  1. I have given the bypass permission ( admin level permission ) to all groups and users.
  2. Whenever I run the main.py the docker image of the custom_connector is not running as I have check with docker ps command.
  3. Can you help me to correct main.py. Complete log is given here. 
    {"timestamp": "2024-07-25T03:41:47.224911Z", "level": "INFO", "name": "api", "message": "Health check (platform version)..."}
{"timestamp": "2024-07-25T03:41:47.322209Z", "level": "INFO", "name": "api", "message": "Health check (platform version)..."}
{"timestamp": "2024-07-25T03:41:47.523379Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Connector registered with ID", "attributes": {"id": "2f3558fc-6eb7-413f-9ae0-5cab8a38cbab"}}
{"timestamp": "2024-07-25T03:41:47.523814Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Starting PingAlive thread"}
{"timestamp": "2024-07-25T03:41:47.524220Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Starting Common Vulnerabilities and Exposures connector..."}
{"timestamp": "2024-07-25T03:41:47.525336Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Common Vulnerabilities and Exposures connector last run: 2024-07-25 01:03:30"}
{"timestamp": "2024-07-25T03:41:47.525439Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Common Vulnerabilities and Exposures will run!"}
{"timestamp": "2024-07-25T03:41:47.525517Z", "level": "INFO", "name": "api", "message": "Initiate work", "attributes": {"connector_id": "2f3558fc-6eb7-413f-9ae0-5cab8a38cbab"}}
{"timestamp": "2024-07-25T03:41:47.562601Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "1 STIX2 objects have been compiled by Common Vulnerabilities and Exposures connector. "}
{"timestamp": "2024-07-25T03:41:47.562883Z", "level": "INFO", "name": "api", "message": "Initiate work", "attributes": {"connector_id": "2f3558fc-6eb7-413f-9ae0-5cab8a38cbab"}}
{"timestamp": "2024-07-25T03:41:47.594740Z", "level": "INFO", "name": "api", "message": "Update action expectations", "attributes": {"work_id": "work_2f3558fc-6eb7-413f-9ae0-5cab8a38cbab_2024-07-25T03:41:47.572Z", "expectations": 1}}
{"timestamp": "2024-07-25T03:41:47.636161Z", "level": "ERROR", "name": "pika.adapters.utils.selector_ioloop_adapter", "message": "Address resolution failed: gaierror(-3, 'Temporary failure in name resolution')"}
{"timestamp": "2024-07-25T03:41:47.637000Z", "level": "ERROR", "name": "pika.adapters.utils.connection_workflow", "message": "getaddrinfo failed: gaierror(-3, 'Temporary failure in name resolution')."}
{"timestamp": "2024-07-25T03:41:47.638141Z", "level": "ERROR", "name": "pika.adapters.utils.connection_workflow", "message": "AMQP connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-3, 'Temporary failure in name resolution'); first exception - None."}
{"timestamp": "2024-07-25T03:41:47.638449Z", "level": "ERROR", "name": "pika.adapters.utils.connection_workflow", "message": "AMQPConnectionWorkflow - reporting failure: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-3, 'Temporary failure in name resolution'); first exception - None"}
{"timestamp": "2024-07-25T03:41:47.638732Z", "level": "ERROR", "name": "pika.adapters.blocking_connection", "message": "Connection workflow failed: AMQPConnectionWorkflowFailed: 1 exceptions in all; last exception - gaierror(-3, 'Temporary failure in name resolution'); first exception - None"}
{"timestamp": "2024-07-25T03:41:47.641183Z", "level": "ERROR", "name": "pika.adapters.blocking_connection", "message": "Error in _create_connection().", "exc_info": "Traceback (most recent call last):\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py\", line 451, in _create_connection\n    raise self._reap_last_connection_workflow_error(error)\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/utils/selector_ioloop_adapter.py\", line 565, in _resolve\n    result = socket.getaddrinfo(self._host, self._port, self._family,\n  File \"/usr/lib/python3.10/socket.py\", line 955, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -3] Temporary failure in name resolution"}
{"timestamp": "2024-07-25T03:41:47.643043Z", "level": "ERROR", "name": "Common Vulnerabilities and Exposures", "message": "[Errno -3] Temporary failure in name resolution", "exc_info": "Traceback (most recent call last):\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/src/lib/external_import.py\", line 132, in run\n    bundle_objects = self._collect_intelligence()\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/main.py\", line 80, in _collect_intelligence\n    self.helper.send_stix2_bundle(\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pycti/connector/opencti_connector_helper.py\", line 1306, in send_stix2_bundle\n    pika_connection = pika.BlockingConnection(pika_parameters)\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py\", line 360, in __init__\n    self._impl = self._create_connection(parameters, _impl_class)\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/blocking_connection.py\", line 451, in _create_connection\n    raise self._reap_last_connection_workflow_error(error)\n  File \"/home/devuser/openCTI_trial2/connectors/external-import/custom_connector/.venv/lib/python3.10/site-packages/pika/adapters/utils/selector_ioloop_adapter.py\", line 565, in _resolve\n    result = socket.getaddrinfo(self._host, self._port, self._family,\n  File \"/usr/lib/python3.10/socket.py\", line 955, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\nsocket.gaierror: [Errno -3] Temporary failure in name resolution"}
{"timestamp": "2024-07-25T03:41:47.643379Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Common Vulnerabilities and Exposures connector successfully run, storing last_run as 1721878907"}
{"timestamp": "2024-07-25T03:41:47.643649Z", "level": "INFO", "name": "api", "message": "Reporting work update_processed", "attributes": {"work_id": "work_2f3558fc-6eb7-413f-9ae0-5cab8a38cbab_2024-07-25T03:41:47.532Z"}}
{"timestamp": "2024-07-25T03:41:47.676006Z", "level": "INFO", "name": "Common Vulnerabilities and Exposures", "message": "Last_run stored, next run in: 0.0 hours"}
romain-filigran commented 3 months ago

Hello @meetghodasara: It seems that your connector is not able to connect to RabbitMQ. First option, try running your “connector container” in the same docker context as OpenCTI (same docker-compose.yml). Second option: you need to expose the rabbitmq port and add DNS resolution to resolve “rabbitmq”.

meetghodasara commented 3 months ago

Thanks you @romain-filigran .

I am tried these steps but still unable to resolve the issue.

  1. Using the custom connector docker compose , I have build image of the custom connector.
  2. I have add that image as the docker compose of Docker Setup for OpenCTI
  3. After re-start the docker compose of OpenCTI , I got same error as here IMPORTANT (https://github.com/OpenCTI-Platform/docker/issues/81) Screenshot 2024-07-25 at 10 33 22 PM
  4. Here is the log of custom connector running on docker.

my_connector | {"timestamp": "2024-07-25T09:37:18.220076Z", "level": "ERROR", "name": "api", "message": "HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused'))", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/usr/local/lib/python3.11/http/client.py\", line 1298, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.11/http/client.py\", line 1058, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.11/http/client.py\", line 996, in send\n self.connect()\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 403, in health_check\n test = self.query(\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 336, in query\n r = self.session.post(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 637, in post\n return self.request(\"POST\", url, data=data, json=json, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused'))"}

  1. When I run the main.py of the connector after docker compose of OpenCTI. I get the given below error. OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration.

Here is the connection error log given below -

{"timestamp": "2024-07-25T09:07:37.539232Z", "level": "INFO", "name": "api", "message": "Health check (platform version)..."} {"timestamp": "2024-07-25T09:07:37.559077Z", "level": "ERROR", "name": "api", "message": "HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused'))", "exc_info": "Traceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/usr/lib/python3.10/http/client.py\", line 1278, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib/python3.10/http/client.py\", line 1038, in _send_output\n self.send(msg)\n File \"/usr/lib/python3.10/http/client.py\", line 976, in send\n self.connect()\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/pycti/api/opencti_api_client.py\", line 403, in health_check\n test = self.query(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/pycti/api/opencti_api_client.py\", line 336, in query\n r = self.session.post(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/sessions.py\", line 637, in post\n return self.request(\"POST\", url, data=data, json=json, **kwargs)\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused'))"}

romain-filigran commented 3 months ago

Can you share your docker configuration of the connector. The error is different from the previous one. If your connector is now in the same docker context, you certainly need to change the “OPENCTI_URL” variable in your connector configuration to point to: http://opencti:8080 instead of http://localhost:8080

meetghodasara commented 3 months ago

Here is the docker configuration of the connector. 

version: '3'
services:
  myconnector:
    build: .
    container_name: myconnector
    environment:
      - CONNECTOR_NAME=myconnector
      - CONNECTOR_SCOPE=stix2
      - OPENCTI_URL=http://opencti:8080
      - OPENCTI_TOKEN=e801b101-ef00-4e24-9593-1d32911bace9
      - CONNECTOR_ID=1c5fb53b-75fb-43fd-8d40-bcefc1ea9a2a
      - CONNECTOR_CONFIDENCE_LEVEL=100
      - CONNECTOR_LOG_LEVEL=info
      - CONNECTOR_RUN_EVERY=60s
    restart: always

Using this command, I have build an docker image. docker build -t opencti/connector-myconnector .

Here is the complete docker-compose file

version: '3'
services:
  redis:
    image: redis:7.2.5
    restart: always
    volumes:
      - redisdata:/data
    networks:
      - docker_default
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.13.4
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      # Comment-out the line below for a cluster of multiple nodes
      - discovery.type=single-node
      # Uncomment the line below below for a cluster of multiple nodes
      # - cluster.name=docker-cluster
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - thread_pool.search.queue_size=5000
      - logger.org.elasticsearch.discovery="ERROR"
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    networks:
      - docker_default
  minio:
    image: minio/minio:RELEASE.2024-05-28T17-19-04Z # Use "minio/minio:RELEASE.2024-05-28T17-19-04Z-cpuv1" to troubleshoot compatibility issues with CPU
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}    
    command: server /data
    restart: always
    networks:
      - docker_default
  rabbitmq:
    image: rabbitmq:3.13-management
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ_NODENAME=rabbit01@localhost
    volumes:
      - amqpdata:/var/lib/rabbitmq
    networks:
      - docker_default
    restart: always
  opencti:
    container_name: opencti
    image: opencti/platform:6.2.7
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=8080
      - APP__BASE_URL=${OPENCTI_BASE_URL}
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__PORT=25
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
    ports:
      - "8080:8080"
    depends_on:
      - redis
      - elasticsearch
      - minio
      - rabbitmq
    restart: always
    networks:
    - docker_default
  worker:
    image: opencti/worker:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=info
    depends_on:
      - opencti
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-export-file-stix:
    image: opencti/connector-export-file-stix:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/json
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default
  connector-export-file-csv:
    image: opencti/connector-export-file-csv:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileCsv
      - CONNECTOR_SCOPE=text/csv
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default
  connector-export-file-txt:
    image: opencti/connector-export-file-txt:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileTxt
      - CONNECTOR_SCOPE=text/plain
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default
  connector-import-file-stix:
    image: opencti/connector-import-file-stix:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportFileStix
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/json,text/xml
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default
  connector-import-document:
    image: opencti/connector-import-document:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportDocument
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/pdf,text/plain,text/html
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - IMPORT_DOCUMENT_CREATE_INDICATOR=true
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default
  connector-analysis:
    image: opencti/connector-import-document:6.2.7
    environment:
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_ANALYSIS_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_ANALYSIS
      - CONNECTOR_NAME=ImportDocumentAnalysis
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=false # Validate any bundle before import
      - CONNECTOR_SCOPE=application/pdf,text/plain,text/html
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default
  myconnector:
    image : opencti/connector-myconnector
    environment:
      - CONNECTOR_NAME=${CONNECTOR_NAME}
      - CONNECTOR_SCOPE=${CONNECTOR_SCOPE}
      - OPENCTI_URL=${OPENCTI_BASE_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${MY_CONNECTOR_ID}
      - CONNECTOR_CONFIDENCE_LEVEL=${CONNECTOR_CONFIDENCE_LEVEL}
      - CONNECTOR_LOG_LEVEL=${CONNECTOR_LOG_LEVEL}
      - CONNECTOR_RUN_EVERY=${CONNECTOR_RUN_EVERY}
    restart: always
    depends_on:
      - opencti
    networks:
      - docker_default

volumes:
  esdata:
  s3data:
  redisdata:
  amqpdata:

networks:
  docker_default:
    external: true

Here is the .env file

OPENCTI_ADMIN_EMAIL=admin@opencti.io
OPENCTI_ADMIN_PASSWORD=ChangeMePlease
OPENCTI_ADMIN_TOKEN=e801b101-ef00-4e24-9593-1d32911bace9
OPENCTI_BASE_URL=http://opencti:8080
MINIO_ROOT_USER=e796f86f-6189-4bc9-ad4c-4423851dd273
MINIO_ROOT_PASSWORD=3a6a478f-67df-416f-8840-a36c433669b2
RABBITMQ_DEFAULT_USER=guest
RABBITMQ_DEFAULT_PASS=guest
ELASTIC_MEMORY_SIZE=6G
CONNECTOR_HISTORY_ID=8016ce47-0852-4b99-93de-25cb8850544d
CONNECTOR_EXPORT_FILE_STIX_ID=a4beb427-811f-47ab-bb2c-0f2ea8351e43
CONNECTOR_EXPORT_FILE_CSV_ID=77d635ce-2559-4a50-8255-63d22098f457
CONNECTOR_IMPORT_FILE_STIX_ID=d7c477ca-6798-47a0-ae0d-fcaa2af9a0e9
CONNECTOR_EXPORT_FILE_TXT_ID=02142d62-7639-4c72-b369-b70bfa898637
CONNECTOR_IMPORT_DOCUMENT_ID=cb03c0fd-945f-42ef-9113-26b5b5505f74
CONNECTOR_ANALYSIS_ID=cb07c0fd-945f-42ef-9113-26b5b5505f74
MY_CONNECTOR_ID=1c5fb53b-75fb-43fd-8d40-bcefc1ea9a2a
SMTP_HOSTNAME=localhost
CONNECTOR_CONFIDENCE_LEVEL=100
CONNECTOR_LOG_LEVEL=info
CONNECTOR_RUN_EVERY=60s
CONNECTOR_UPDATE_EXISTING_DATA=false
CONNECTOR_SCOPE=stix2
CONNECTOR_NAME=myconnector
EXTRA_PARAMETER=foobar
CONNECTOR_ID=2f3558fc-6eb7-413f-9ae0-5cab8a38cbab

After this all, I have used the below command to run OpenCTI. docker-compose up --build -d

Here is the logs for both OpenCTI and myconnector

{"category":"APP","cause":{"_error":{},"_showLocations":false,"_showPath":false,"data":{"cause":{"meta":{"body":null,"headers":null,"meta":{"aborted":false,"attempts":3,"connection":{"_openRequests":0,"deadCount":4,"headers":{},"id":"http://elasticsearch:9200/","resurrectTimeout":1722077077774,"roles":{"data":true,"ingest":true},"status":"dead","url":"http://elasticsearch:9200/"},"context":null,"name":"opensearch-js","request":{"id":1,"options":{},"params":{"body":null,"headers":{"user-agent":"opensearch-js/2.8.0 (linux 5.15.0-117-generic-x64; Node.js v20.15.1)"},"method":"GET","path":"/","querystring":"","timeout":30000}}},"statusCode":null},"name":"ConnectionError"},"genre":"TECHNICAL","http_status":500},"internalData":{},"name":"CONFIGURATION_ERROR","time_thrown":"2024-07-27T10:36:37.775Z"},"level":"error","message":"[OPENCTI] System dependencies check failed","source":"backend","timestamp":"2024-07-27T10:36:37.775Z","version":"6.2.7"}
{"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Search engine seems down","name":"CONFIGURATION_ERROR","stack":"CONFIGURATION_ERROR: Search engine seems down\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at ConfigurationError (/opt/opencti/build/src/config/errors.js:76:53)\n    at /opt/opencti/build/src/database/engine.js:230:15\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at searchEngineVersion (/opt/opencti/build/src/database/engine.js:226:22)\n    at searchEngineInit (/opt/opencti/build/src/database/engine.js:312:27)\n    at checkSystemDependencies (/opt/opencti/build/src/initialization.js:32:3)\n    at platformStart (/opt/opencti/build/src/boot.js:14:7)"},{"message":"connect ECONNREFUSED 172.17.0.4:9200","name":"ConnectionError","stack":"ConnectionError: connect ECONNREFUSED 172.17.0.4:9200\n    at ClientRequest.onError (/opt/opencti/build/node_modules/@opensearch-project/opensearch/lib/Connection.js:129:16)\n    at ClientRequest.emit (node:events:519:28)\n    at Socket.socketErrorListener (node:_http_client:500:9)\n    at Socket.emit (node:events:519:28)\n    at emitErrorNT (node:internal/streams/destroy:169:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"}],"level":"error","message":"Search engine seems down","source":"backend","timestamp":"2024-07-27T10:36:37.787Z","version":"6.2.7"}

Myconnector log

{"timestamp": "2024-07-27T10:36:33.303302Z", "level": "INFO", "name": "api", "message": "Health check (platform version)..."}
{"timestamp": "2024-07-27T10:36:33.312895Z", "level": "ERROR", "name": "api", "message": "HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))", "exc_info": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 196, in _new_conn\n    sock = connection.create_connection(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n    raise err\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n    sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n    response = self._make_request(\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n    conn.request(\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 398, in request\n    self.endheaders()\n  File \"/usr/local/lib/python3.11/http/client.py\", line 1298, in endheaders\n    self._send_output(message_body, encode_chunked=encode_chunked)\n  File \"/usr/local/lib/python3.11/http/client.py\", line 1058, in _send_output\n    self.send(msg)\n  File \"/usr/local/lib/python3.11/http/client.py\", line 996, in send\n    self.connect()\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 236, in connect\n    self.sock = self._new_conn()\n                ^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 211, in _new_conn\n    raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n           ^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 403, in health_check\n    test = self.query(\n           ^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 336, in query\n    r = self.session.post(\n        ^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 637, in post\n    return self.request(\"POST\", url, data=data, json=json, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))"}
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
romain-filigran commented 2 months ago

it seems that the elasticsearch container does not start ("Search engine seems down"). Please check its logs

meetghodasara commented 2 months ago

First of all Thank you so much @romain-filigran and OpenCTI community.

Yes it was Search engine seems down. As I was normally docker-compose command so it wasn't solved but with Portainer. It is solved. Seems like some network problem was there.

I have completed solved issue #2398 with the following steps.

Pre-configuration

  1. Set the VM size for Elastic search with the given command sudo sysctl -w vm.max_map_count=1048575
  2. Every UUID should be generated from the UUID Generator website
  3. Use Docker Swarm and Portainer to configuration of OpenCTI.

Installation steps

  1. I have did the installation of Docker, Creation of Docker Swarm , and configuration of Portainer from the given article. Also here is the video tutorial for the same.

  2. Custom connector configuration is as follow. Here is the docker-compose.yaml

    
opencti:
url: "http://opencti:8080"
token: 396187c1-82c7-4fae-bb99-21e1591eb02e

connector: id: 2f3558fc-6eb7-413f-9ae0-5cab8a38cbab type: 'EXTERNAL_IMPORT' name: 'myconnector' scope: 'identity,attack-pattern,course-of-action,intrusion-set,malware,tool,report,vulnerability,campaign,incident,indicator,infrastructure,location,note,threat-actor,tool,software' run_and_terminate: false log_level: 'info'

myconnector:

with using `docker-compose build` for building of docker image. 

#### Portainer Stack Configuration 

3. The given below is docker-compose for the OpenCTI. 
IMPORTANT > [docker compose](https://github.com/meetghodasara/docker/blob/master/docker-compose.yml) 

4. The given below is .env file for the OpenCTI docker-compose. (Note: these uuid generated from [UUID Generator website](https://www.uuidgenerator.net/) )

OPENCTI_ADMIN_EMAIL=admin@opencti.io OPENCTI_ADMIN_PASSWORD=Admin1234 OPENCTI_ADMIN_TOKEN=396187c1-82c7-4fae-bb99-21e1591eb02e OPENCTI_BASE_URL=http://opencti:8080 MINIO_ROOT_USER=7f15a361-25e0-4c45-b065-e46c7a8c85a2 MINIO_ROOT_PASSWORD=c754d202-6afe-4f78-b1a8-a3a455001daa RABBITMQ_DEFAULT_USER=guest RABBITMQ_DEFAULT_PASS=guest CONNECTOR_EXPORT_FILE_STIX_ID=dd817c8b-abae-460a-9ebc-97b1551e70e6 CONNECTOR_EXPORT_FILE_CSV_ID=7ba187fb-fde8-4063-92b5-c3da34060dd7 CONNECTOR_EXPORT_FILE_TXT_ID=ca715d9c-bd64-4351-91db-33a8d728a58b CONNECTOR_IMPORT_FILE_STIX_ID=72327164-0b35-482b-b5d6-a5a3f76b845f CONNECTOR_IMPORT_DOCUMENT_ID=c3970f8a-ce4b-4497-a381-20b7256f56f0 CONNECTOR_ANALYSIS_ID=4dffd77c-ec11-4abe-bca7-fd997f79fa36 ELASTIC_MEMORY_SIZE=4G SMTP_HOSTNAME=localhost



5. Here is the our custom connector is running successfully with the data shown in dashboard.

Your Custom connector can be found here Data>Ingestion>Connector>myconnector.
<img width="621" alt="Screenshot 2024-07-29 at 3 23 41 PM" src="https://github.com/user-attachments/assets/73b6c3a7-13c1-4df2-b463-da5d5b571b8d">

Sample data shown in dashboard that are injected through myconnector. 
<img width="619" alt="Screenshot 2024-07-29 at 3 24 35 PM" src="https://github.com/user-attachments/assets/80614993-789b-400b-a0ed-b7103e9359bc">

### Other issues I had got during setup that are solved with the above method. 

1. Elastic Search seems down in OpenCTI Platform container.  | (https://github.com/OpenCTI-Platform/opencti/issues/1412)
2. Data not displayed in Dashboard after Docker Installation of OpenCTI. | (https://github.com/OpenCTI-Platform/docker/issues/81)
3. RabbitMQ Pika Connection error. AMPQ Connection workflow failed. | (https://github.com/OpenCTI-Platform/connectors/issues/54)
4. GraphQL Http Connection Error. Failed to establish a new connection, connection refused | (https://github.com/OpenCTI-Platform/opencti/issues/155)
5. OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration... | (https://github.com/OpenCTI-Platform/opencti/issues/5159)
pesquisa1234 commented 2 months ago

I am using version 6.2.6 and used this guy video you sent.

I am getting this is the error with alienvault connector: @meetghodasara

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 403, in health_check test = self.query( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 336, in query r = self.session.post( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6dfb88c50>: Failed to establish a new connection: [Errno 111] Connection refused')) Terminated Terminated

meetghodasara commented 2 months ago

Could you please share your docker-compose and .env file? Also please share the log of OpenCTI platform container.

I have written one article about Installation of OpenCTI using Docker so you can refer it too. Installing OpenCTI with Docker: A Step-by-Step Guide

You can refer the my docker compose and .env.sample file from the given Docker-Github

nino-filigran commented 2 months ago

Reading this thread makes me understand that issue is solved. I'm closing this ticket, but feel free to re-open it if needed.