RabbitMQ Connection fails when trying local developped connector

[mohanex]

Environment

1. OS : Ubuntu Server on PROXMOX with 32Go of ram and 14 CPUs
2. OpenCTI version: 5.12.17 Entreprise edition

Description

I'm new to OpenCTI and to better understand how it works, I developed a very basic external-import connector. The problem appears when I debug the connector locally. see the image below : Data fetching and transformation to STIX2 bundle works perfectly, also on my OpenCTI platform I can see that my connector got successfully some information but didn't manage to send them to the plateforme. see the image below : Keep in mind that the connector is ran in the same machine where the Docker hosting OpenCTI is running.

Steps to produce the Bug

1. Install OpenCTI version: 5.12.17 on Docker using portainer and the feault given docker.compose file (https://github.com/OpenCTI-Platform/docker/blob/95c1b6298a71a9962b0ae0091ebf6c4b1860728e/docker-compose.yml)
2. Follow connector developping steps (https://docs.opencti.io/latest/development/connectors/)
3. When I arrive to "Running the connector" segment, I get the RabbitMQ error.

[Jipegien]

@helene-nguyen and @Megafredo can you help me for qualifying this ?

[mohanex]

Hello,

To give you information about reproduction of the bug: 1 - clone my repo containing the connector (https://github.com/mohanex/OpenCTI_Connector) into a virtual machine containing docker that hosts an OpenCTI instance. 2 - Change Tokens and OpenCTI URl if necessary 3 - Change PROXY setting if necessary 4 - Run the connector in local using python3 or for example VScode Python Debugger.

If you still don't have enough information, tell me about things you want to know.

[helene-nguyen]

Thanks, I will check it and give you an update @mohanex

[helene-nguyen]

Hi @mohanex, I'm on it, I reproduce the bug with the following steps:

• OpenCTI instance running on a Docker container
• Connector running locally

The RabbitMQ works fine and I also get the error message:

"Error sending bundle: [Errno 11001] getaddrinfo failed"

Means we need to resolve the host. I'll give you an update as soon as possible :)

[mohanex]

Hello,

Thank you for the reproduction. If you refer to my first screenshot, I also get this "getaddrinfo failed" then followed with RabbitmQ errors like the "AMQP connection workflow failed"... Do you get the same ? And when you say RabbitMQ works fine did it get to connect with the connector ?

[helene-nguyen]

Hello, Yes, same error :

RabbitMQ works fine in the container, I can access the GUI:

[mohanex]

Hi,

Yep I have exactly the same issue, RabbitMQ is also working fine for me with the other imported connectors from docker hub images. I will try to import my connector into docker and see if it works there. I'll keep you updated

[helene-nguyen]

Hello,

After some investigations and tests, I've found a way for you to run your OpenCTI in a container and the connector aside.

Running the connector locally and connecting it with an OpenCTI instance in a Docker container will give you this error because you'll need all the context (RabbitMQ, worker) too.

A workaround for this is to run your connector in a container and add a network, let say opencti-network, which you can add to the end of your docker-compose.yml used to deploy the OpenCTI instance:

networks:
  opencti-network:
    driver: bridge

And you can add it for each services. Here's the docker-compose.yml used for the test:

version: '3'
services:
  redis:
    image: redis:7.2.4
    volumes:
      - redisdata1:/data
    networks:
      - opencti-network

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.12.0
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    environment:
      # Comment-out the line below for a cluster of multiple nodes
      - discovery.type=single-node
      # Uncomment the line below below for a cluster of multiple nodes
      # - cluster.name=docker-cluster
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    ports:
      - 9201:9200
      - 9301:9300
    networks:
      - opencti-network

  minio:
    image: minio/minio:RELEASE.2024-01-16T16-07-38Z
    volumes:
      - s3data1:/data
    ports:
      - "9003:9000"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}    
    command: server /data
    networks:
      - opencti-network

  rabbitmq:
    image: rabbitmq:3.12-management
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ_NODENAME=rabbit01@localhost
    volumes:
      - amqpdata1:/var/lib/rabbitmq
    ports:
      - 5673:5672
      - 15673:15672
    networks:
      - opencti-network

  opencti:
    image: opencti/platform:5.12.20
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=8081
      - APP__BASE_URL=${OPENCTI_BASE_URL}
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__PORT=25
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
    ports:
      - "8081:8081"
    depends_on:
      - redis
      - elasticsearch
      - minio
      - rabbitmq
    networks:
      - opencti-network

  worker:
    image: opencti/worker:5.12.20
    environment:
      - OPENCTI_URL=http://opencti:8081
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=info
    depends_on:
      - opencti
    deploy:
      mode: replicated
      replicas: 2
    networks:
      - opencti-network

networks:
  opencti-network:
    driver: bridge

volumes:
  esdata1:
  s3data1:
  redisdata1:
  amqpdata1:

Once containers are running, and OpenCTI works:

You can check networks with:

docker network ls

That way, you can find the name of the network where all containers are connected.

To connect your connector to this network, you have to add:

networks:
  docker_opencti-network:
    external: true

And add it to your service. Here your docker-compose.yml file:

version: '3'
services:
  connector-blacklistip:
    build: .
    environment:
      # Connector's definition parameters:
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=${CONNECTOR_NAME}
      - CONNECTOR_SCOPE=Incident
      # Connector's generic execution parameters:
      - OPENCTI_URL=http://opencti:8081
      - OPENCTI_TOKEN=${OPENCTI_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_ID}
      - CONNECTOR_CONFIDENCE_LEVEL=${CONNECTOR_CONFIDENCE_LEVEL} # From 0 (Unknown) to 100 (Fully trusted).
      - CONNECTOR_LOG_LEVEL=${CONNECTOR_LOG_LEVEL}
      - CONNECTOR_RUN_EVERY=${CONNECTOR_RUN_EVERY}
      # Connector's custom execution parameters:
      - EXTRA_PARAMETER=${EXTRA_PARAMETER}
      # Add proxy parametrs below if you need them
      # - HTTP_PROXY=ChangeMe
      # - HTTPS_PROXY=ChangeMe
      # - NO_PROXY=ChangeMe\
    networks:
      - docker_opencti-network

networks:
  docker_opencti-network:
    external: true

And when running your container, no more errors:

That way, you can test your code and your connector :)

I close the issue but we can re-open it if needed!

Hope that helps!

PS: About the name of the connector, could you rename it in DenyListIp as it is a more descriptive name?

[mohanex]

Hello,

Thank you for the detailed response. I was about to post a comment to explain the same approach as you did. However I still don't see the feeds and data in my OpenCTI Dashboard, did you get to see them with your configuration ?

[helene-nguyen]

I don't see the feed at the moment, I think there may be a piece of code that needs to be changed for OpenCTI to ingest the data.

Here are some documentation that can help you:

• OpenCTI Connectors Documentation

[mohanex]

Thank you very much, I appreciate your help. I will be looking for the feed problem and report it here whenever I manage to find the solution.