Data Connectors not Running

[OpenSecureCo]

Description

Hey Team, I am noticing that not all my data connectors are running even though they are not displaying errors and all related services are up and running. I am using portainer to manage the opencti stack and running the 5.2.1 images. It is odd because some connectors are fully working, such as AlienVault and MISP, but other connectors such as AM!TT, RISKIQ and others never run.

The UI states that they are "In progress" but they have been in this state for a few weeks now and looking at the logs of the respective containers I see no errors:

Even the opencti connector never runs:

Below is my docker-compose file:

version: '3'
services:
  opencti-dev-redis:
    container_name: opencti-dev-redis
    image: redis:6.2.6
    privileged: true
    deploy:
      placement:
        constraints:
          - node.role == worker
    restart: unless-stopped
    volumes:
      - redisdata:/data
    ports:
      - 6379:6379
  opencti-dev-elasticsearch:
    container_name: opencti-dev-elasticsearch
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0
    privileged: true
    deploy:
      placement:
        constraints:
          - node.role == manager
      resources:
        limits:
          memory: 4000M
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node

      - xpack.ml.enabled=false
    ports:
      - "9200:9200"
      - "9300:9300"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
  opencti-dev-kibana:
    container_name: opencti-dev-kibana
    image: docker.elastic.co/kibana/kibana:7.17.0
    environment:
      - ELASTICSEARCH_HOSTS=http://opencti-dev-elasticsearch:9200
    restart: unless-stopped
    ports:
      - 5601:5601
    depends_on:
      - opencti-dev-elasticsearch
  opencti-dev-minio:
    container_name: opencti-dev-minio
    image: minio/minio:RELEASE.2022-02-05T04-40-59Z

    privileged: true
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
      - "9001:9001"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}    
    command: server /data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    restart: always
  opencti-dev-rabbitmq:
    container_name: opencti-dev-rabbitmq
    image: rabbitmq:3.9-management
    privileged: true
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
    ports:
      - 5672:5672
      - 15672:15672
  opencti:
    image: opencti/platform:5.2.1
    privileged: true
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=4000
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - REDIS__HOSTNAME=opencti-dev-redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://opencti-dev-elasticsearch:9200
      - MINIO__ENDPOINT=opencti-dev-minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=opencti-dev-rabbitmq
      - RABBITMQ__PORT=5672
      - RABBITMQ__PORT_MANAGEMENT=15672
      - RABBITMQ__MANAGEMENT_SSL=false
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - SMTP__HOSTNAME=${SMTP_HOSTNAME}
      - SMTP__PORT=25
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
    ports:
      - "4000:4000"
    depends_on:
      - opencti-dev-redis
      - opencti-dev-elasticsearch
      - opencti-dev-minio
      - opencti-dev-rabbitmq
    restart: always
  worker:
    image: opencti/worker:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=info
    depends_on:
      - opencti
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-history:
    image: opencti/connector-history:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_HISTORY_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=STREAM
      - CONNECTOR_NAME=History
      - CONNECTOR_SCOPE=history
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-stix:
    image: opencti/connector-export-file-stix:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/json
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-csv:
    image: opencti/connector-export-file-csv:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_CSV_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileCsv
      - CONNECTOR_SCOPE=text/csv
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-export-file-txt:
    image: opencti/connector-export-file-txt:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_EXPORT_FILE_TXT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_EXPORT_FILE
      - CONNECTOR_NAME=ExportFileTxt
      - CONNECTOR_SCOPE=text/plain
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-file-stix:
    image: opencti/connector-import-file-stix:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_FILE_STIX_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportFileStix
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/json,text/xml
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
    restart: always
    depends_on:
      - opencti
  connector-import-document:
    image: opencti/connector-import-document:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=${CONNECTOR_IMPORT_DOCUMENT_ID} # Valid UUIDv4
      - CONNECTOR_TYPE=INTERNAL_IMPORT_FILE
      - CONNECTOR_NAME=ImportDocument
      - CONNECTOR_VALIDATE_BEFORE_IMPORT=true # Validate any bundle before import
      - CONNECTOR_SCOPE=application/pdf,text/plain,text/html
      - CONNECTOR_AUTO=true # Enable/disable auto-import of file
      - CONNECTOR_ONLY_CONTEXTUAL=false # Only extract data related to an entity (a report, a threat actor, etc.)
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - IMPORT_DOCUMENT_CREATE_INDICATOR=true
    restart: always
    depends_on:
      - opencti
  connector-misp:
    image: opencti/connector-misp:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=cfa1b34c-4f07-4920-bd39-5b162eb8ae6e
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=MISP
      - CONNECTOR_SCOPE=misp
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=false
      - CONNECTOR_LOG_LEVEL=info
      - MISP_URL=redacted
      - MISP_KEY=redacted
      - MISP_SSL_VERIFY=False # Required
      - MISP_DATETIME_ATTRIBUTE=timestamp # Required, filter to be used in query for new MISP events
      - MISP_CREATE_REPORTS=True # Required, create report for MISP event
      - MISP_CREATE_INDICATORS=True # Required, create indicators from attributes
      - MISP_CREATE_OBSERVABLES=True # Required, create observables from attributes
      - MISP_CREATE_OBJECT_OBSERVABLES=True # Required, create text observables for MISP objects
      - MISP_REPORT_CLASS=MISP Event # Optional, report_class if creating report for event
      - MISP_IMPORT_FROM_DATE=2020-01-01 # Optional, import all event from this date
      - MISP_IMPORT_TAGS=opencti:import,type:osint # Optional, list of tags used for import events
      - MISP_IMPORT_TAGS_NOT= # Optional, list of tags to not include
      - MISP_IMPORT_CREATOR_ORGS= # Optional, only import events created by this ORG (put the identifier here)
      - MISP_IMPORT_OWNER_ORGS= # Optional, only import events owned by this ORG (put the identifier here)
      - MISP_IMPORT_DISTRIBUTION_LEVELS=0,1,2,3 # Optional, only import events with the given distribution levels
      - MISP_IMPORT_THREAT_LEVELS=1,2,3,4 # Optional only import events with the given threat levels
      - MISP_IMPORT_ONLY_PUBLISHED=False
      - MISP_IMPORT_WITH_ATTACHMENTS=False # Optional, try to import a PDF file from the attachment attribute
      - MISP_IMPORT_TO_IDS_NO_SCORE=40 # Optional, use as a score for the indicator/observable if the attribute to_ids is no
      - MISP_IMPORT_UNSUPPORTED_OBSERVABLES_AS_TEXT=False #  Optional, import unsupported observable as x_opencti_text
      - MISP_INTERVAL=60 # Required, in minutes
    restart: always
    depends_on:
      - opencti
  connector-thehive:
    image: opencti/connector-thehive:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=36c525df-0671-4b44-a66b-af1d52fb6f10
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=TheHive
      - CONNECTOR_SCOPE=thehive
      - CONNECTOR_CONFIDENCE_LEVEL=80 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_LOG_LEVEL=info
      - THEHIVE_URL=redacted
      - THEHIVE_API_KEY=redacted
      - THEHIVE_CHECK_SSL=false
      - THEHIVE_ORGANIZATION_NAME=SOCFortress
      - THEHIVE_IMPORT_FROM_DATE=2021-01-01T00:00:00 # Optional
    restart: always
    depends_on:
      - opencti
  connector-mitre:
    image: opencti/connector-mitre:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=8aceb1a5-1892-4bdc-a36e-23afa260379e
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - "CONNECTOR_NAME=MITRE ATT&CK"
      - CONNECTOR_SCOPE=marking-definition,identity,attack-pattern,course-of-action,intrusion-set,campaign,malware,tool,report,external-reference-as-report
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_RUN_AND_TERMINATE=false
      - CONNECTOR_LOG_LEVEL=info
      - MITRE_ENTERPRISE_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/enterprise-attack/enterprise-attack.json
      - MITRE_PRE_ATTACK_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/pre-attack/pre-attack.json
      - MITRE_MOBILE_ATTACK_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/mobile-attack/mobile-attack.json
      - MITRE_ICS_ATTACK_FILE_URL=https://raw.githubusercontent.com/mitre/cti/master/ics-attack/ics-attack.json
      - MITRE_INTERVAL=2 # In days, must be strictly greater than 1
    restart: always
    depends_on:
      - opencti
  connector-virustotal:
    image: opencti/connector-virustotal:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=95b526d7-64c7-4353-9c2f-bd31ce9c427f
      - CONNECTOR_TYPE=INTERNAL_ENRICHMENT
      - CONNECTOR_NAME=VirusTotal
      - CONNECTOR_SCOPE=StixFile,Artifact
      - CONNECTOR_AUTO=true # Enable/disable auto-enrichment of observables
      - CONNECTOR_CONFIDENCE_LEVEL=50 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - VIRUSTOTAL_TOKEN=redacted
      - VIRUSTOTAL_MAX_TLP=TLP:AMBER
    restart: always
    depends_on:
      - opencti
  connector-cybercrimetracker:
    image: opencti/connector-cybercrime-tracker:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=9fe0e6f6-623a-4f4f-97b1-168688535abe
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=Cybercrime-Tracker
      - CONNECTOR_SCOPE=cybercrime-tracker
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_LOG_LEVEL=info
      - CYBERCRIME_TRACKER_FEED_URL=http://cybercrime-tracker.net/rss.xml
      - CYBERCRIME_TRACKER_TLP=WHITE
      - CYBERCRIME_TRACKER_INTERVAL=86400
      - CYBERCRIME_TRACKER_CREATE_INDICATORS=true
      - CYBERCRIME_TRACKER_CREATE_OBSERVABLES=true
    restart: always
    depends_on:
      - opencti
  connector-malware-bazaar-recent-additions:
    image: opencti/connector-malwarebazaar-recent-additions:5.2.1
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=MalwareBazaar_Recent_Additions
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - "CONNECTOR_NAME=MalwareBazaar Recent Additions"
      - CONNECTOR_CONFIDENCE_LEVEL=50 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_LOG_LEVEL=info
      - MALWAREBAZAAR_RECENT_ADDITIONS_API_URL=https://mb-api.abuse.ch/api/v1/
      - MALWAREBAZAAR_RECENT_ADDITIONS_COOLDOWN_SECONDS=300 # Time to wait in seconds between subsequent requests
      - MALWAREBAZAAR_RECENT_ADDITIONS_INCLUDE_TAGS=exe,dll,docm,docx,doc,xls,xlsx,xlsm,js # (Optional) Only download files if any tag matches. (Comma separated)
      - MALWAREBAZAAR_RECENT_ADDITIONS_INCLUDE_REPORTERS=joe_sandbox # (Optional) Only download files uploaded by these reporters. (Comma separated)
      - MALWAREBAZAAR_RECENT_ADDITIONS_LABELS=malware-bazar # (Optional) Labels to apply to uploaded Artifacts. (Comma separated)
      - MALWAREBAZAAR_RECENT_ADDITIONS_LABELS_COLOR=#54483b # Color to use for labels
    restart: always
    depends_on:
      - opencti
  connector-opencti:
    image: opencti/connector-opencti:5.2.1
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=aa40a088-2c26-40eb-a801-063b3680ca5c
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=OpenCTI
      - CONNECTOR_SCOPE=marking-definition,identity,location
      - CONNECTOR_CONFIDENCE_LEVEL=90 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_RUN_AND_TERMINATE=false
      - CONNECTOR_LOG_LEVEL=info
      - CONFIG_SECTORS_FILE_URL=https://raw.githubusercontent.com/OpenCTI-Platform/datasets/master/data/sectors.json
      - CONFIG_GEOGRAPHY_FILE_URL=https://raw.githubusercontent.com/OpenCTI-Platform/datasets/master/data/geography.json
      - CONFIG_INTERVAL=2 # In days, must be strictly greater than 1
    restart: always
    depends_on:
      - opencti
  connector-riskiq:
    image: opencti/connector-riskiq:5.2.1
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=fcee7b02-fdbe-4374-9bdd-e23f2ce1b8ef
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=RISKIQ
      - CONNECTOR_SCOPE=riskiq
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - RISKIQ_BASE_URL=https://api.riskiq.net/pt/v2
      - RISKIQ_USER=redacted
      - RISKIQ_PASSWORD=redacted
      - RISKIQ_INTERVAL_SEC=30
    restart: always
    depends_on:
      - opencti
  connector-cve:
    image: opencti/connector-cve:5.2.1
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=4834d3f6-b141-4838-bf10-45088ee6af4d
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=Common Vulnerabilities and Exposures
      - CONNECTOR_SCOPE=identity,vulnerability
      - CONNECTOR_CONFIDENCE_LEVEL=75 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_RUN_AND_TERMINATE=false
      - CONNECTOR_LOG_LEVEL=info
      - CVE_IMPORT_HISTORY=true # Import history at the first run (after only recent), reset the connector state if you want to re-import
      - CVE_NVD_DATA_FEED=https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.json.gz
      - CVE_HISTORY_DATA_FEED=https://nvd.nist.gov/feeds/json/cve/1.1/
      - CVE_INTERVAL=2 # In days, must be strictly greater than 1
    restart: always
    depends_on:
      - opencti
  connector-amitt:
    image: opencti/connector-amitt:5.2.1
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=010c8e95-874b-47d4-8ab6-434b16b66861
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=AM!TT
      - CONNECTOR_SCOPE=identity,attack-pattern,course-of-action,intrusion-set,malware,tool,report
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=true
      - CONNECTOR_LOG_LEVEL=info
      - AMITT_FILE_URL=https://raw.githubusercontent.com/VVX7/cti/master/amitt-attack/amitt-attack.json
      - PRE_AMITT_FILE_URL=https://raw.githubusercontent.com/VVX7/cti/master/amitt-pre-attack/amitt-pre-attack.json
      - AMITT_INTERVAL=2 # In days, must be strictly greater than 1
    restart: always
    depends_on:
      - opencti
  connector-abuseipdb:
    image: opencti/connector-abuseipdb:5.2.1
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=c0601eb7-b94d-42b1-a446-f5dda452cbad
      - CONNECTOR_TYPE=INTERNAL_ENRICHMENT
      - CONNECTOR_NAME=AbuseIPDB
      - CONNECTOR_SCOPE=IPv4-Addr
      - CONNECTOR_AUTO=true
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_LOG_LEVEL=info
      - ABUSEIPDB_API_KEY=redacted
      - ABUSEIPDB_MAX_TLP=TLP:AMBER
    restart: always
  connector-alienvault:
    image: opencti/connector-alienvault:5.2.1
    privileged: true
    environment:
      - OPENCTI_URL=http://opencti:4000
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - CONNECTOR_ID=981d17db-b86b-48ae-b965-14fa26f678b5
      - CONNECTOR_TYPE=EXTERNAL_IMPORT
      - CONNECTOR_NAME=AlienVault
      - CONNECTOR_SCOPE=alienvault
      - CONNECTOR_CONFIDENCE_LEVEL=15 # From 0 (Unknown) to 100 (Fully trusted)
      - CONNECTOR_UPDATE_EXISTING_DATA=false
      - CONNECTOR_LOG_LEVEL=info
      - ALIENVAULT_BASE_URL=https://otx.alienvault.com
      - ALIENVAULT_API_KEY=redacted
      - ALIENVAULT_TLP=White
      - ALIENVAULT_CREATE_OBSERVABLES=true
      - ALIENVAULT_CREATE_INDICATORS=true
      - ALIENVAULT_PULSE_START_TIMESTAMP=2020-05-01T00:00:00                  # BEWARE! Could be a lot of pulses!
      - ALIENVAULT_REPORT_TYPE=threat-report
      - ALIENVAULT_REPORT_STATUS=New
      - ALIENVAULT_GUESS_MALWARE=false                                        # Use tags to guess malware.
      - ALIENVAULT_GUESS_CVE=false                                            # Use tags to guess CVE.
      - ALIENVAULT_EXCLUDED_PULSE_INDICATOR_TYPES=FileHash-MD5,FileHash-SHA1  # Excluded Pulse indicator types.
      - ALIENVAULT_ENABLE_RELATIONSHIPS=true                                  # Enable/Disable relationship creation between SDOs.
      - ALIENVAULT_ENABLE_ATTACK_PATTERNS_INDICATES=true                      # Enable/Disable "indicates" relationships between indicators and attack patterns
      - ALIENVAULT_INTERVAL_SEC=1800
    restart: always
    depends_on:
      - opencti
volumes:
  esdata:
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=redacted,rw
      device: ":/mnt/opencti/es_data/"
  s3data:
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=redacted,rw
      device: ":/mnt/opencti/s3_data/"     
  redisdata:
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=redacted,rw
      device: ":/mnt/opencti/redis_data/"
  amqpdata:
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=redacted,rw
      device: ":/mnt/opencti/amqp_data/"

I have been scratching my head for the last few weeks now and I cannot figure out what the issue may be.

Any insight would be greatly appreciated.

Thanks, Taylor

[00willo]

This is basically, what I'd been seeing, and asking questions on in the luatix slack.

docker exec -it opencti_rabbitmq_1 /bin/bash then run rabbitmqadmin list queues vhost node name messages are there messages on the queue related to alienvault?

You can see which push queue is alienvault's from this command rabbitmqadmin -f long -d 3 list queue | less, then search for alienvalut in the config.scope. It may provide some answers or clues.

I was noticing that sometimes, the stix bundles in the queue to be processed would disappear. This would happen on redeploying the stack's docker-compose, which I was using portainer to manage. But I'd have the same result, "In progress" on the connector, that would never move, like the reference to get a query jobs from that queue had been lost somewhere.

I have more testing to do on stopping containers at times to see if I can repeat some of this behaviour.

[OpenSecureCo]

Was related to the MTU settings of my NIC...the MTU was set to a value of 1450 (set by cloud provider) and docker by default sets an MTU of 1500. I changed my default route to the internet to the NIC holding the boxes public IP (MTU of 1500), restarted and connectors look good now.