Update issue and Failed to resolve 'opencti' ([Errno -2] Name does not resolve)")) when launching connector

eddiestaffy08 commented 6 months ago

Prerequisites

[x] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
[x] I went through old GitHub issues and couldn't find anything relevant
[x] I googled the issue and didn't find anything relevant

Description

I have just tried updating to OpenCTI v6.0.8 and I am unable to reconfigure any of my connectors. I have updated using the docker-compose yml file and changed all of the env variables to match my initial OpenCTI configuration, but during the update I've lost all of my users.

I then tried to just create new users and configure my original connectors (with the updated openCTI version and new user OpenCTI token in the portainer stacks) but now I keep getting an error that states "Failed to resolve 'opencti' ([Errno -2] Name does not resolve)"))".

I had asked a question in Slack about the correct way to launch my update to the newest version but I did not get any replies.

Environment

OS (where OpenCTI server runs): Debian 12.4
OpenCTI version: 6.0.8
OpenCTI client: python
Other environment details: OpenCTI has been deployed using portainer and docker swarm

Reproducible Steps

Steps to create the smallest reproducible scenario:

Stopped OpenCTI and connector stacks
Created new stacks to preserve the stopped stacks in case of update issues, but used all the corresponding UUIDs, tokens, and env variables.
launched the updated OpenCTI stack.
Error caused me to need to manually create the opencti_backend network in my control panel, once I did this, the updated OpenCTI version launched but it had no information in it, including no users so I then couldn't reconfigure my connectors.

Additional information

OPENCTI UPDATE CONFIG:

version: '3'
services:
  redis:
    networks: 
      - backend
    image: redis:7.2.4
    restart: always
    volumes:
      - redisdata:/data
  elasticsearch:
    networks: 
      - backend 
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      # Comment-out the line below for a cluster of multiple nodes
      - discovery.type=single-node
      # Uncomment the line below below for a cluster of multiple nodes
      # - cluster.name=docker-cluster
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - thread_pool.search.queue_size=5000
      - logger.org.elasticsearch.discovery="ERROR"
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
  minio:
    networks: 
      - backend
    image: minio/minio:RELEASE.2024-01-16T16-07-38Z
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
#    deploy:
#      labels:
#       - "traefik.enable=true"
#       - "traefik.http.routers.minio.entrypoints=https"
#       - "traefik.http.routers.minio.rule=Host(`minio.juddy.ovh`)"
#       - "traefik.http.routers.minio.service=minio"
#       - "traefik.http.routers.minio.tls=true"
#       - "traefik.http.services.minio.loadbalancer.server.port=9000"
#       - "traefik.http.routers.minio.tls.certresolver=mytlschallenge"
#       - "traefik.http.routers.minio_http.entrypoints=http"
#       - "traefik.http.routers.minio_http.rule=Host(`minio.juddy.ovh`)"
#       - "traefik.http.routers.minio_http.middlewares=traefik-redirectscheme"
#       - "traefik.http.middlewares.traefik-redirectscheme.redirectscheme.scheme=https"      
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}    
    command: server /data
    restart: always
  rabbitmq:
    networks: 
      - backend
    image: rabbitmq:3.13-management
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ_NODENAME=rabbit01@localhost
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
  opencti:
    networks: 
      - backend
      - traefik-public
    deploy:
      labels:
**labels redacted**  
    image: opencti/platform:6.0.8
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=4000
      - APP__BASE_URL=${OPENCTI_BASE_URL}
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__PORT=25
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy

    depends_on:
      - redis
      - elasticsearch
      - minio
      - rabbitmq
    restart: always
  worker:
    networks: 
      - backend
    image: opencti/worker:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=info
    depends_on:
      - opencti
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-export-file-stix:
    networks: 
      - backend
    image: opencti/connector-export-file-stix:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/json
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-csv:
    networks: 
      - backend
    image: opencti/connector-export-file-csv:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileCsv
      - CONNECTOR_SCOPE=text/csv
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-txt:
    networks: 
      - backend  
    image: opencti/connector-export-file-txt:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileTxt
      - CONNECTOR_SCOPE=text/plain
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-file-stix:
    networks: 
      - backend  
    image: opencti/connector-import-file-stix:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportFileStix
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/json,text/xml
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-document:
    networks: 
      - backend  
    image: opencti/connector-import-document:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportDocument
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/pdf,text/plain,text/html
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - IMPORT_DOCUMENT_CREATE_INDICATOR=true
    restart: always
    depends_on:
      - opencti

volumes:
  esdata:
  s3data:
  redisdata:
  amqpdata:

networks: 
  traefik-public: 
    external: true
  backend:`

CONNECTOR CONFIG:

version: '3'
services:
  connector-alienvault:
    networks:
      - opencti_backend
    image: opencti/connector-alienvault:6.0.8
    environment:
      - OPENCTI_URL=${OPENCTI_URL}
      - OPENCTI_TOKEN=${OPENCTI_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_ID}
     # - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=AlienVault
      - CONNECTOR_SCOPE=alienvault
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=false
      - CONNECTOR_LOG_LEVEL=error
      - ALIENVAULT_BASE_URL=https://otx.alienvault.com
      - ALIENVAULT_API_KEY=${ALIENVAULT_API_KEY}
      - ALIENVAULT_TLP=White
      - ALIENVAULT_CREATE_OBSERVABLES=true
      - ALIENVAULT_CREATE_INDICATORS=true
      - ALIENVAULT_PULSE_START_TIMESTAMP=2022-05-01T00:00:00                  # BEWARE! Could be a lot of pulses!
      - ALIENVAULT_REPORT_TYPE=threat-report
      - ALIENVAULT_REPORT_STATUS=New
      - ALIENVAULT_GUESS_MALWARE=false                                        # Use tags to guess malware.
      - ALIENVAULT_GUESS_CVE=false                                            # Use tags to guess CVE.
      - ALIENVAULT_EXCLUDED_PULSE_INDICATOR_TYPES=FileHash-MD5,FileHash-SHA1  # Excluded Pulse indicator types.
      - ALIENVAULT_ENABLE_RELATIONSHIPS=true                                  # Enable/Disable relationship creation between SDOs.
      - ALIENVAULT_ENABLE_ATTACK_PATTERNS_INDICATES=false                     # Enable/Disable "indicates" relationships between indicators and attack patterns
      - ALIENVAULT_INTERVAL_SEC=1800
    restart: always

networks:
  opencti_backend:
    external: true`

PORTAINER LOG:

`ERROR HTTPConnectionPool(host='opencti', port=4000): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f8748580ed0>: Failed to resolve 'opencti' ([Errno -2] Name does not resolve)")) | timestamp=2024-04-03T05:42:15.202900Z name=api exc_info=Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/socket.py", line 962, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
socket.gaierror: [Errno -2] Name does not resolve

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/usr/local/lib/python3.11/http/client.py", line 1293, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1052, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 990, in send
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 238, in connect
    self.sock = self._new_conn()
                ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in _new_conn
    raise NameResolutionError(self.host, self, e) from e
urllib3.exceptions.NameResolutionError: <urllib3.connection.HTTPConnection object at 0x7f8748580ed0>: Failed to resolve 'opencti' ([Errno -2] Name does not resolve)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='opencti', port=4000): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f8748580ed0>: Failed to resolve 'opencti' ([Errno -2] Name does not resolve)"))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 387, in health_check
    test = self.query(
           ^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pycti/api/opencti_api_client.py", line 328, in query
    r = self.session.post(
        ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 637, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='opencti', port=4000): Max retries exceeded with url: /graphql (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f8748580ed0>: Failed to resolve 'opencti' ([Errno -2] Name does not resolve)"))

Terminated`

AmmisettyBhuvanesh commented 6 months ago

I am also facing the same issue...i am getting every same error for every connector to the portainer

gazsoimi commented 6 months ago

are you using the old docker-compose, or the new native docker compose built in plugin?

I had a similar problem just now, and came across this as searching.

according the logs you provided, the problem is that docker-compose (which is using oython) is unable to communicate with the docker socket, and cant resolve the name for the opencti.

you can mess around with the python install https://github.com/google-deepmind/alphafold/issues/867 this might still work,

but as the old standalone docker-compose is deprecated for almost a year now,

I recommend to add the docker repo, and update for a newer docker, with native docker-compose-plugin.

https://docs.docker.com/compose/install/linux/#install-using-the-repository

https://docs.docker.com/compose/migrate/

after this, it worked for me without a problem.

eddiestaffy08 commented 6 months ago

Hi @gazsoimi, thanks for your comment! I have just followed the repository update instructions for docker compose but am still faced with the same error unfortunately.

AmmisettyBhuvanesh commented 6 months ago

hi @gazsoimi and @eddiestaffy08 I am about to update that compose ....i will reply soon .

AmmisettyBhuvanesh commented 6 months ago

i have updated native docker compose plugin but still i am getting same error for every other connector(Alienvault,opencti,misp)