Closed meetghodasara closed 2 months ago
Hello @meetghodasara: It seems that your connector is not able to connect to RabbitMQ. First option, try running your “connector container” in the same docker context as OpenCTI (same docker-compose.yml). Second option: you need to expose the rabbitmq port and add DNS resolution to resolve “rabbitmq”.
Thanks you @romain-filigran .
I am tried these steps but still unable to resolve the issue.
my_connector | {"timestamp": "2024-07-25T09:37:18.220076Z", "level": "ERROR", "name": "api", "message": "HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused'))", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/usr/local/lib/python3.11/http/client.py\", line 1298, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.11/http/client.py\", line 1058, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.11/http/client.py\", line 996, in send\n self.connect()\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 403, in health_check\n test = self.query(\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 336, in query\n r = self.session.post(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 637, in post\n return self.request(\"POST\", url, data=data, json=json, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=80): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f6724267890>: Failed to establish a new connection: [Errno 111] Connection refused'))"}
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration.
Here is the connection error log given below -
{"timestamp": "2024-07-25T09:07:37.539232Z", "level": "INFO", "name": "api", "message": "Health check (platform version)..."} {"timestamp": "2024-07-25T09:07:37.559077Z", "level": "ERROR", "name": "api", "message": "HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused'))", "exc_info": "Traceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/usr/lib/python3.10/http/client.py\", line 1278, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/lib/python3.10/http/client.py\", line 1038, in _send_output\n self.send(msg)\n File \"/usr/lib/python3.10/http/client.py\", line 976, in send\n self.connect()\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/pycti/api/opencti_api_client.py\", line 403, in health_check\n test = self.query(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/pycti/api/opencti_api_client.py\", line 336, in query\n r = self.session.post(\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/sessions.py\", line 637, in post\n return self.request(\"POST\", url, data=data, json=json, **kwargs)\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n File \"/home/devuser/openCTI_trial2/connectors/external-import/myconnector/.venv/lib/python3.10/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f7365658d30>: Failed to establish a new connection: [Errno 111] Connection refused'))"}
Can you share your docker configuration of the connector. The error is different from the previous one. If your connector is now in the same docker context, you certainly need to change the “OPENCTI_URL” variable in your connector configuration to point to: http://opencti:8080 instead of http://localhost:8080
Here is the docker configuration of the connector.
version: '3'
services:
myconnector:
build: .
container_name: myconnector
environment:
- CONNECTOR_NAME=myconnector
- CONNECTOR_SCOPE=stix2
- OPENCTI_URL=http://opencti:8080
- OPENCTI_TOKEN=e801b101-ef00-4e24-9593-1d32911bace9
- CONNECTOR_ID=1c5fb53b-75fb-43fd-8d40-bcefc1ea9a2a
- CONNECTOR_CONFIDENCE_LEVEL=100
- CONNECTOR_LOG_LEVEL=info
- CONNECTOR_RUN_EVERY=60s
restart: always
Using this command, I have build an docker image.
docker build -t opencti/connector-myconnector .
Here is the complete docker-compose file
version: '3'
services:
redis:
image: redis:7.2.5
restart: always
volumes:
- redisdata:/data
networks:
- docker_default
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.13.4
volumes:
- esdata:/usr/share/elasticsearch/data
environment:
# Comment-out the line below for a cluster of multiple nodes
- discovery.type=single-node
# Uncomment the line below below for a cluster of multiple nodes
# - cluster.name=docker-cluster
- xpack.ml.enabled=false
- xpack.security.enabled=false
- thread_pool.search.queue_size=5000
- logger.org.elasticsearch.discovery="ERROR"
- "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
restart: always
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
networks:
- docker_default
minio:
image: minio/minio:RELEASE.2024-05-28T17-19-04Z # Use "minio/minio:RELEASE.2024-05-28T17-19-04Z-cpuv1" to troubleshoot compatibility issues with CPU
volumes:
- s3data:/data
ports:
- "9000:9000"
environment:
MINIO_ROOT_USER: ${MINIO_ROOT_USER}
MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
command: server /data
restart: always
networks:
- docker_default
rabbitmq:
image: rabbitmq:3.13-management
environment:
- RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
- RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
- RABBITMQ_NODENAME=rabbit01@localhost
volumes:
- amqpdata:/var/lib/rabbitmq
networks:
- docker_default
restart: always
opencti:
container_name: opencti
image: opencti/platform:6.2.7
environment:
- NODE_OPTIONS=--max-old-space-size=8096
- APP__PORT=8080
- APP__BASE_URL=${OPENCTI_BASE_URL}
- APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
- APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
- APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
- APP__APP_LOGS__LOGS_LEVEL=error
- REDIS__HOSTNAME=redis
- REDIS__PORT=6379
- ELASTICSEARCH__URL=http://elasticsearch:9200
- MINIO__ENDPOINT=minio
- MINIO__PORT=9000
- MINIO__USE_SSL=false
- MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
- MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
- RABBITMQ__HOSTNAME=rabbitmq
- RABBITMQ__PORT=5672
- RABBITMQ__PORT_MANAGEMENT=15672
- RABBITMQ__MANAGEMENT_SSL=false
- RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
- RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
- SMTP__HOSTNAME=${SMTP_HOSTNAME}
- SMTP__PORT=25
- PROVIDERS__LOCAL__STRATEGY=LocalStrategy
ports:
- "8080:8080"
depends_on:
- redis
- elasticsearch
- minio
- rabbitmq
restart: always
networks:
- docker_default
worker:
image: opencti/worker:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- WORKER_LOG_LEVEL=info
depends_on:
- opencti
deploy:
mode: replicated
replicas: 3
restart: always
connector-export-file-stix:
image: opencti/connector-export-file-stix:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
- CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
- CONNECTOR_NAME=ExportFileStix2
- CONNECTOR_SCOPE=application/json
- CONNECTOR_LOG_LEVEL=info
restart: always
depends_on:
- opencti
networks:
- docker_default
connector-export-file-csv:
image: opencti/connector-export-file-csv:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
- CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
- CONNECTOR_NAME=ExportFileCsv
- CONNECTOR_SCOPE=text/csv
- CONNECTOR_LOG_LEVEL=info
restart: always
depends_on:
- opencti
networks:
- docker_default
connector-export-file-txt:
image: opencti/connector-export-file-txt:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
- CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
- CONNECTOR_NAME=ExportFileTxt
- CONNECTOR_SCOPE=text/plain
- CONNECTOR_LOG_LEVEL=info
restart: always
depends_on:
- opencti
networks:
- docker_default
connector-import-file-stix:
image: opencti/connector-import-file-stix:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
- CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
- CONNECTOR_NAME=ImportFileStix
- CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
- CONNECTOR_SCOPE=application/json,text/xml
- CONNECTOR_AUTO=true # Enable/disable auto-import of file
- CONNECTOR_LOG_LEVEL=info
restart: always
depends_on:
- opencti
networks:
- docker_default
connector-import-document:
image: opencti/connector-import-document:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
- CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
- CONNECTOR_NAME=ImportDocument
- CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
- CONNECTOR_SCOPE=application/pdf,text/plain,text/html
- CONNECTOR_AUTO=true # Enable/disable auto-import of file
- CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
- CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
- CONNECTOR_LOG_LEVEL=info
- IMPORT_DOCUMENT_CREATE_INDICATOR=true
restart: always
depends_on:
- opencti
networks:
- docker_default
connector-analysis:
image: opencti/connector-import-document:6.2.7
environment:
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${CONNECTOR_ANALYSIS_ID} # Valid UUIDv4
- CONNECTOR_TYPE=INTERNAL_ANALYSIS
- CONNECTOR_NAME=ImportDocumentAnalysis
- CONNECTOR_VALIDATE_BEFORE_IMPORT=false # Validate any bundle before import
- CONNECTOR_SCOPE=application/pdf,text/plain,text/html
- CONNECTOR_AUTO=true # Enable/disable auto-import of file
- CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
- CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
- CONNECTOR_LOG_LEVEL=info
restart: always
depends_on:
- opencti
networks:
- docker_default
myconnector:
image : opencti/connector-myconnector
environment:
- CONNECTOR_NAME=${CONNECTOR_NAME}
- CONNECTOR_SCOPE=${CONNECTOR_SCOPE}
- OPENCTI_URL=${OPENCTI_BASE_URL}
- OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
- CONNECTOR_ID=${MY_CONNECTOR_ID}
- CONNECTOR_CONFIDENCE_LEVEL=${CONNECTOR_CONFIDENCE_LEVEL}
- CONNECTOR_LOG_LEVEL=${CONNECTOR_LOG_LEVEL}
- CONNECTOR_RUN_EVERY=${CONNECTOR_RUN_EVERY}
restart: always
depends_on:
- opencti
networks:
- docker_default
volumes:
esdata:
s3data:
redisdata:
amqpdata:
networks:
docker_default:
external: true
Here is the .env file
OPENCTI_ADMIN_EMAIL=admin@opencti.io
OPENCTI_ADMIN_PASSWORD=ChangeMePlease
OPENCTI_ADMIN_TOKEN=e801b101-ef00-4e24-9593-1d32911bace9
OPENCTI_BASE_URL=http://opencti:8080
MINIO_ROOT_USER=e796f86f-6189-4bc9-ad4c-4423851dd273
MINIO_ROOT_PASSWORD=3a6a478f-67df-416f-8840-a36c433669b2
RABBITMQ_DEFAULT_USER=guest
RABBITMQ_DEFAULT_PASS=guest
ELASTIC_MEMORY_SIZE=6G
CONNECTOR_HISTORY_ID=8016ce47-0852-4b99-93de-25cb8850544d
CONNECTOR_EXPORT_FILE_STIX_ID=a4beb427-811f-47ab-bb2c-0f2ea8351e43
CONNECTOR_EXPORT_FILE_CSV_ID=77d635ce-2559-4a50-8255-63d22098f457
CONNECTOR_IMPORT_FILE_STIX_ID=d7c477ca-6798-47a0-ae0d-fcaa2af9a0e9
CONNECTOR_EXPORT_FILE_TXT_ID=02142d62-7639-4c72-b369-b70bfa898637
CONNECTOR_IMPORT_DOCUMENT_ID=cb03c0fd-945f-42ef-9113-26b5b5505f74
CONNECTOR_ANALYSIS_ID=cb07c0fd-945f-42ef-9113-26b5b5505f74
MY_CONNECTOR_ID=1c5fb53b-75fb-43fd-8d40-bcefc1ea9a2a
SMTP_HOSTNAME=localhost
CONNECTOR_CONFIDENCE_LEVEL=100
CONNECTOR_LOG_LEVEL=info
CONNECTOR_RUN_EVERY=60s
CONNECTOR_UPDATE_EXISTING_DATA=false
CONNECTOR_SCOPE=stix2
CONNECTOR_NAME=myconnector
EXTRA_PARAMETER=foobar
CONNECTOR_ID=2f3558fc-6eb7-413f-9ae0-5cab8a38cbab
After this all, I have used the below command to run OpenCTI.
docker-compose up --build -d
Here is the logs for both OpenCTI and myconnector
{"category":"APP","cause":{"_error":{},"_showLocations":false,"_showPath":false,"data":{"cause":{"meta":{"body":null,"headers":null,"meta":{"aborted":false,"attempts":3,"connection":{"_openRequests":0,"deadCount":4,"headers":{},"id":"http://elasticsearch:9200/","resurrectTimeout":1722077077774,"roles":{"data":true,"ingest":true},"status":"dead","url":"http://elasticsearch:9200/"},"context":null,"name":"opensearch-js","request":{"id":1,"options":{},"params":{"body":null,"headers":{"user-agent":"opensearch-js/2.8.0 (linux 5.15.0-117-generic-x64; Node.js v20.15.1)"},"method":"GET","path":"/","querystring":"","timeout":30000}}},"statusCode":null},"name":"ConnectionError"},"genre":"TECHNICAL","http_status":500},"internalData":{},"name":"CONFIGURATION_ERROR","time_thrown":"2024-07-27T10:36:37.775Z"},"level":"error","message":"[OPENCTI] System dependencies check failed","source":"backend","timestamp":"2024-07-27T10:36:37.775Z","version":"6.2.7"}
{"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Search engine seems down","name":"CONFIGURATION_ERROR","stack":"CONFIGURATION_ERROR: Search engine seems down\n at error (/opt/opencti/build/src/config/errors.js:8:10)\n at ConfigurationError (/opt/opencti/build/src/config/errors.js:76:53)\n at /opt/opencti/build/src/database/engine.js:230:15\n at processTicksAndRejections (node:internal/process/task_queues:95:5)\n at searchEngineVersion (/opt/opencti/build/src/database/engine.js:226:22)\n at searchEngineInit (/opt/opencti/build/src/database/engine.js:312:27)\n at checkSystemDependencies (/opt/opencti/build/src/initialization.js:32:3)\n at platformStart (/opt/opencti/build/src/boot.js:14:7)"},{"message":"connect ECONNREFUSED 172.17.0.4:9200","name":"ConnectionError","stack":"ConnectionError: connect ECONNREFUSED 172.17.0.4:9200\n at ClientRequest.onError (/opt/opencti/build/node_modules/@opensearch-project/opensearch/lib/Connection.js:129:16)\n at ClientRequest.emit (node:events:519:28)\n at Socket.socketErrorListener (node:_http_client:500:9)\n at Socket.emit (node:events:519:28)\n at emitErrorNT (node:internal/streams/destroy:169:8)\n at emitErrorCloseNT (node:internal/streams/destroy:128:3)\n at processTicksAndRejections (node:internal/process/task_queues:82:21)"}],"level":"error","message":"Search engine seems down","source":"backend","timestamp":"2024-07-27T10:36:37.787Z","version":"6.2.7"}
Myconnector log
{"timestamp": "2024-07-27T10:36:33.303302Z", "level": "INFO", "name": "api", "message": "Health check (platform version)..."}
{"timestamp": "2024-07-27T10:36:33.312895Z", "level": "ERROR", "name": "api", "message": "HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))", "exc_info": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 196, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 85, in create_connection\n raise err\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py\", line 73, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 789, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 495, in _make_request\n conn.request(\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 398, in request\n self.endheaders()\n File \"/usr/local/lib/python3.11/http/client.py\", line 1298, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.11/http/client.py\", line 1058, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.11/http/client.py\", line 996, in send\n self.connect()\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 236, in connect\n self.sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NewConnectionError(\nurllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line 843, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py\", line 519, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 403, in health_check\n test = self.query(\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py\", line 336, in query\n r = self.session.post(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 637, in post\n return self.request(\"POST\", url, data=data, json=json, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, **send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fe6e485a8d0>: Failed to establish a new connection: [Errno 111] Connection refused'))"}
OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration...
it seems that the elasticsearch container does not start ("Search engine seems down"). Please check its logs
First of all Thank you so much @romain-filigran and OpenCTI community.
Yes it was Search engine seems down
. As I was normally docker-compose command so it wasn't solved but with Portainer. It is solved. Seems like some network problem was there.
I have completed solved issue #2398 with the following steps.
sudo sysctl -w vm.max_map_count=1048575
I have did the installation of Docker, Creation of Docker Swarm , and configuration of Portainer from the given article. Also here is the video tutorial for the same.
Custom connector configuration is as follow. Here is the docker-compose.yaml
opencti:
url: "http://opencti:8080"
token: 396187c1-82c7-4fae-bb99-21e1591eb02e
connector: id: 2f3558fc-6eb7-413f-9ae0-5cab8a38cbab type: 'EXTERNAL_IMPORT' name: 'myconnector' scope: 'identity,attack-pattern,course-of-action,intrusion-set,malware,tool,report,vulnerability,campaign,incident,indicator,infrastructure,location,note,threat-actor,tool,software' run_and_terminate: false log_level: 'info'
myconnector:
with using `docker-compose build` for building of docker image.
#### Portainer Stack Configuration
3. The given below is docker-compose for the OpenCTI.
IMPORTANT > [docker compose](https://github.com/meetghodasara/docker/blob/master/docker-compose.yml)
4. The given below is .env file for the OpenCTI docker-compose. (Note: these uuid generated from [UUID Generator website](https://www.uuidgenerator.net/) )
OPENCTI_ADMIN_EMAIL=admin@opencti.io OPENCTI_ADMIN_PASSWORD=Admin1234 OPENCTI_ADMIN_TOKEN=396187c1-82c7-4fae-bb99-21e1591eb02e OPENCTI_BASE_URL=http://opencti:8080 MINIO_ROOT_USER=7f15a361-25e0-4c45-b065-e46c7a8c85a2 MINIO_ROOT_PASSWORD=c754d202-6afe-4f78-b1a8-a3a455001daa RABBITMQ_DEFAULT_USER=guest RABBITMQ_DEFAULT_PASS=guest CONNECTOR_EXPORT_FILE_STIX_ID=dd817c8b-abae-460a-9ebc-97b1551e70e6 CONNECTOR_EXPORT_FILE_CSV_ID=7ba187fb-fde8-4063-92b5-c3da34060dd7 CONNECTOR_EXPORT_FILE_TXT_ID=ca715d9c-bd64-4351-91db-33a8d728a58b CONNECTOR_IMPORT_FILE_STIX_ID=72327164-0b35-482b-b5d6-a5a3f76b845f CONNECTOR_IMPORT_DOCUMENT_ID=c3970f8a-ce4b-4497-a381-20b7256f56f0 CONNECTOR_ANALYSIS_ID=4dffd77c-ec11-4abe-bca7-fd997f79fa36 ELASTIC_MEMORY_SIZE=4G SMTP_HOSTNAME=localhost
5. Here is the our custom connector is running successfully with the data shown in dashboard.
Your Custom connector can be found here Data>Ingestion>Connector>myconnector.
<img width="621" alt="Screenshot 2024-07-29 at 3 23 41 PM" src="https://github.com/user-attachments/assets/73b6c3a7-13c1-4df2-b463-da5d5b571b8d">
Sample data shown in dashboard that are injected through myconnector.
<img width="619" alt="Screenshot 2024-07-29 at 3 24 35 PM" src="https://github.com/user-attachments/assets/80614993-789b-400b-a0ed-b7103e9359bc">
### Other issues I had got during setup that are solved with the above method.
1. Elastic Search seems down in OpenCTI Platform container. | (https://github.com/OpenCTI-Platform/opencti/issues/1412)
2. Data not displayed in Dashboard after Docker Installation of OpenCTI. | (https://github.com/OpenCTI-Platform/docker/issues/81)
3. RabbitMQ Pika Connection error. AMPQ Connection workflow failed. | (https://github.com/OpenCTI-Platform/connectors/issues/54)
4. GraphQL Http Connection Error. Failed to establish a new connection, connection refused | (https://github.com/OpenCTI-Platform/opencti/issues/155)
5. OpenCTI API is not reachable. Waiting for OpenCTI API to start or check your configuration... | (https://github.com/OpenCTI-Platform/opencti/issues/5159)
I am using version 6.2.6 and used this guy video you sent.
I am getting this is the error with alienvault connector: @meetghodasara
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 403, in health_check test = self.query( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 336, in query r = self.session.post( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 637, in post return self.request("POST", url, data=data, json=json, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, send_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=8080): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd6dfb88c50>: Failed to establish a new connection: [Errno 111] Connection refused')) Terminated Terminated
Could you please share your docker-compose and .env file? Also please share the log of OpenCTI platform container.
I have written one article about Installation of OpenCTI using Docker so you can refer it too. Installing OpenCTI with Docker: A Step-by-Step Guide
You can refer the my docker compose and .env.sample file from the given Docker-Github
Reading this thread makes me understand that issue is solved. I'm closing this ticket, but feel free to re-open it if needed.
Description
I have create the custom_connector with the docker configuration given below.
Docker configuration.
main.py
Environment
Reproducible Steps
Steps to create the smallest reproducible scenario:
Expected Output
The data should be inserted and displayed in the Observable objects. The dashboard should have observable objects.
Actual Output
Ingestion - connectors shows the activity log but it always show in-progress.
The Observable objects are empty here.
The activity log contains these activity.
Additional information
docker ps
command.