OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.46k stars 948 forks source link

No ingestion, low resource usage, connections reset #8846

Open dominictory opened 3 weeks ago

dominictory commented 3 weeks ago

Description

At present, I have 68k queued bundles, of which <1 are getting processed/sec, as below, which is constant at the moment. In worker and connector logs, whilst some bundles do get processed, there are sporadic connection reset errors, which point to RabbitMQ which is showing as healthy in the stack, as below, with no errors other than the occasional errors below every 5 mins or so. The backend platform does show as unhealthy, with numerous errors, as below at the bottom. Connectors seem to go between active and inactive regularly. I would appreciate some help troubleshooting further please, and especially with Redis/platform if I can try any different configurations. Thanks :)

image

image

image

image

image

image

2024-10-31T11:54:23.631945580Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> closing AMQP connection <0.2750204.0> (172.20.0.43:51702 -> 172.20.0.9:5672):

2024-10-31T11:54:23.632025790Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> missed heartbeats from client, timeout: 10s

My docker-compose setup is as follows:

1x frontend platform (users, internal export/import connectors only, no managers) 1x backend platform (3x workers, connectors, all managers)

VM: Ubuntu 22.04 LTS; 16 cores; 64GB RAM

Current load average: 7.22, 5.52, 4.45 Memory usage: total used free shared buff/cache available Mem: 62Gi 48Gi 4.1Gi 43Mi 10Gi 13Gi Swap: 8.0Gi 5.4Gi 2.6Gi

docker-compose.yml sample:

services:
  redis:
    image: redis:${REDIS_VERSION}
    restart: always
    volumes:
      - redisdata3:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3
  elasticsearch:
    image: elasticsearch:${ELK_VERSION}
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - thread_pool.search.queue_size=5000
      - logger.org.elasticsearch.discovery="ERROR"
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    healthcheck:
      test: curl -s http://elasticsearch:9200 >/dev/null || exit 1
      interval: 30s
      timeout: 10s
      retries: 50
  minio:
    image: minio/minio:${MINIO_VERSION}
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    command: server /data
    restart: always
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 10s
      timeout: 5s
      retries: 3
  rabbitmq:
    image: rabbitmq:${RMQ_VERSION}
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ__CONSUMER_TIMEOUT=36000000
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
    healthcheck:
      test: rabbitmq-diagnostics -q ping
      interval: 30s
      timeout: 30s
      retries: 3
  opencti-front:
    image: opencti/platform:${OPENCTI_VERSION}
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=443
      - APP__BASE_URL=https://localhost
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - APP__APP_LOGS__LOGS_FILES=false
      - APP__SESSION_TIMEOUT=28800000
      - APP__HTTPS_CERT__KEY=/certs/cti.key
      - APP__HTTPS_CERT__CRT=/certs/cti.crt
      - RULE_ENGINE__ENABLED=false
      - HISTORY_MANAGER__ENABLED=false
      - TASK_SCHEDULER__ENABLED=false
      - SYNC_MANAGER__ENABLED=false
      - EXPIRATION_SCHEDULER__ENABLED=false
      - RETENTION_MANAGER__ENABLED=false
      - NOTIFICATION_MANAGER__ENABLED=false
      - PUBLISHER_MANAGER__ENABLED=false
      - INGESTION_MANAGER__ENABLED=false
      - PLAYBOOK_MANAGER__ENABLED=false
      - ACTIVITY_MANAGER__ENABLED=false
      - CONNECTOR_MANAGER__ENABLED=false
      - IMPORT_CSV_BUILT_IN_CONNECTOR__ENABLED=false
      - FILE_INDEX_MANAGER__ENABLED=false
      - INDICATOR_DECAY_MANAGER__ENABLED=false
      - GARBAGE_COLLECTION_MANAGER__ENABLED=false
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ__QUEUE_TYPE="quorum"
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
      - APP__HEALTH_ACCESS_KEY=${OPENCTI_HEALTHCHECK_ACCESS_KEY}
      #- XTM__OPENBAS_URL=http://10.254.203.11:8080
      #- XTM__OPENBAS_TOKEN=803889f7-c97d-4899-b984-4f575297d771
    volumes:
      - tls:/certs
    ports:
      - "443:443"
    restart: always
  opencti-data:
    image: opencti/platform:${OPENCTI_VERSION}
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=8080
      - APP__BASE_URL=http://localhost:8080
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - APP__APP_LOGS__LOGS_FILES=false
      - RULE_ENGINE__ENABLED=false
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ__QUEUE_TYPE="quorum"
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
      - APP__HEALTH_ACCESS_KEY=${OPENCTI_HEALTHCHECK_ACCESS_KEY}
    depends_on:
      redis:
        condition: service_healthy
      elasticsearch:
        condition: service_healthy
      minio:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    ports:
      - "8080:8080"
    restart: always
    healthcheck:
      test:  ["CMD", "wget", "-qO-", "http://opencti-data:8080/health?health_access_key=${OPENCTI_HEALTHCHECK_ACCESS_KEY}"]
      interval: 10s
      timeout: 5s
      retries: 100
  worker:
    image: opencti/worker:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=http://opencti-data:8080
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=error
    depends_on:
      opencti-data:
        condition: service_healthy
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-export-file-stix:
    image: opencti/connector-export-file-stix:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=https://opencti-front
      - OPENCTI_TOKEN=ebf85396-544e-4a78-ab76-b66c5ea0b663
      - CONNECTOR_ID=dd817c8b-abae-460a-9ebc-97b1551e70e6
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/vnd.oasis.stix+json
      - CONNECTOR_LOG_LEVEL=error
    restart: always
  connector-alienvault:
    image: opencti/connector-alienvault:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=http://opencti-data:8080
      - OPENCTI_TOKEN=bd3bb843-cafe-451d-a5a7-711d32494ddb
      - CONNECTOR_ID=c1af30c7-b667-4167-9fcb-693972e61091
      - CONNECTOR_NAME=AlienVault
      - CONNECTOR_SCOPE=alienvault
      - CONNECTOR_LOG_LEVEL=error
      - CONNECTOR_DURATION_PERIOD=PT30M
      - ALIENVAULT_BASE_URL=https://otx.alienvault.com
      - ALIENVAULT_API_KEY=redacted
      - ALIENVAULT_TLP=White
      - ALIENVAULT_CREATE_OBSERVABLES=true
      - ALIENVAULT_CREATE_INDICATORS=true
      - ALIENVAULT_PULSE_START_TIMESTAMP=2024-01-01T00:00:00
      - ALIENVAULT_REPORT_TYPE=threat-report
      - ALIENVAULT_REPORT_STATUS=New
      - ALIENVAULT_GUESS_MALWARE=true
      - ALIENVAULT_GUESS_CVE=true
      - ALIENVAULT_EXCLUDED_PULSE_INDICATOR_TYPES=FileHash-MD5,FileHash-SHA1
      - ALIENVAULT_ENABLE_RELATIONSHIPS=true
      - ALIENVAULT_ENABLE_ATTACK_PATTERNS_INDICATES=true
    restart: always
...
volumes:
  esdata:
    external: true
    name: opencti_esdata
  s3data:
    external: true
    name: opencti_s3data
  redisdata3:
    external: true
    name: opencti_redisdata3
  amqpdata:
    external: true
    name: opencti_amqpdata
  tls:
    external: true
    name: opencti_tls

.env:

OPENCTI_ADMIN_EMAIL=admin@opencti.ct
OPENCTI_ADMIN_PASSWORD=@;0.Sx>{G5).Mzn#^VvU
OPENCTI_ADMIN_TOKEN=4e95798b-21a6-4571-819f-d974f2c2abea
OPENCTI_VERSION=6.3.6
REDIS_VERSION=7.4.0
ELK_VERSION=8.15.2
ELASTIC_MEMORY_SIZE=32G
MINIO_VERSION=RELEASE.2024-05-28T17-19-04Z
RMQ_VERSION=3.13-management
MINIO_ROOT_USER=opencti
MINIO_ROOT_PASSWORD=OMYPQ5skRD8WU6v^iIRL
RABBITMQ_DEFAULT_USER=opencti
RABBITMQ_DEFAULT_PASS=T+9ix{l{9>U<i?KZ>@B#
OPENCTI_HEALTHCHECK_ACCESS_KEY=dccc470f-7339-4026-a7cd-2ab6c82a480c

RabbitMQ log sample:

2024-10-31T11:53:21.165303199Z 2024-10-31 11:53:21.164784+00:00 [info] <0.2864758.0> accepting AMQP connection <0.2864758.0> (172.20.0.54:35830 -> 172.20.0.9:5672)

2024-10-31T11:53:21.168424077Z 2024-10-31 11:53:21.168055+00:00 [info] <0.2864758.0> connection <0.2864758.0> (172.20.0.54:35830 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:54:23.631945580Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> closing AMQP connection <0.2750204.0> (172.20.0.43:51702 -> 172.20.0.9:5672):

2024-10-31T11:54:23.632025790Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> missed heartbeats from client, timeout: 10s

2024-10-31T11:55:18.580620899Z 2024-10-31 11:55:18.580178+00:00 [info] <0.2867390.0> accepting AMQP connection <0.2867390.0> (172.20.0.17:35972 -> 172.20.0.9:5672)

2024-10-31T11:55:18.580770860Z 2024-10-31 11:55:18.580435+00:00 [info] <0.2867388.0> accepting AMQP connection <0.2867388.0> (172.20.0.17:35982 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581007160Z 2024-10-31 11:55:18.580566+00:00 [info] <0.2867380.0> accepting AMQP connection <0.2867380.0> (172.20.0.17:35992 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581423563Z 2024-10-31 11:55:18.580647+00:00 [info] <0.2867396.0> accepting AMQP connection <0.2867396.0> (172.20.0.17:36008 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581579732Z 2024-10-31 11:55:18.580829+00:00 [info] <0.2867384.0> accepting AMQP connection <0.2867384.0> (172.20.0.17:36018 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581645892Z 2024-10-31 11:55:18.581004+00:00 [info] <0.2867381.0> accepting AMQP connection <0.2867381.0> (172.20.0.17:36034 -> 172.20.0.9:5672)

2024-10-31T11:55:18.582195331Z 2024-10-31 11:55:18.581093+00:00 [info] <0.2867400.0> accepting AMQP connection <0.2867400.0> (172.20.0.17:36044 -> 172.20.0.9:5672)

2024-10-31T11:55:18.582210797Z 2024-10-31 11:55:18.581270+00:00 [info] <0.2867402.0> accepting AMQP connection <0.2867402.0> (172.20.0.17:36056 -> 172.20.0.9:5672)

2024-10-31T11:55:18.628130363Z 2024-10-31 11:55:18.627511+00:00 [info] <0.2867380.0> connection <0.2867380.0> (172.20.0.17:35992 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628178378Z 2024-10-31 11:55:18.627577+00:00 [info] <0.2867402.0> connection <0.2867402.0> (172.20.0.17:36056 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628237012Z 2024-10-31 11:55:18.627640+00:00 [info] <0.2867381.0> connection <0.2867381.0> (172.20.0.17:36034 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628426966Z 2024-10-31 11:55:18.627773+00:00 [info] <0.2867384.0> connection <0.2867384.0> (172.20.0.17:36018 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628548824Z 2024-10-31 11:55:18.627652+00:00 [info] <0.2867396.0> connection <0.2867396.0> (172.20.0.17:36008 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628614181Z 2024-10-31 11:55:18.627894+00:00 [info] <0.2867388.0> connection <0.2867388.0> (172.20.0.17:35982 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.635481580Z 2024-10-31 11:55:18.635034+00:00 [info] <0.2867400.0> connection <0.2867400.0> (172.20.0.17:36044 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.635547523Z 2024-10-31 11:55:18.635117+00:00 [info] <0.2867390.0> connection <0.2867390.0> (172.20.0.17:35972 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.658688709Z 2024-10-31 11:55:18.657703+00:00 [info] <0.2867402.0> closing AMQP connection <0.2867402.0> (172.20.0.17:36056 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.673803618Z 2024-10-31 11:55:18.673165+00:00 [info] <0.2867381.0> closing AMQP connection <0.2867381.0> (172.20.0.17:36034 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.673849465Z 2024-10-31 11:55:18.673174+00:00 [info] <0.2867380.0> closing AMQP connection <0.2867380.0> (172.20.0.17:35992 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.685619113Z 2024-10-31 11:55:18.684949+00:00 [info] <0.2867384.0> closing AMQP connection <0.2867384.0> (172.20.0.17:36018 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.685677895Z 2024-10-31 11:55:18.685143+00:00 [info] <0.2867388.0> closing AMQP connection <0.2867388.0> (172.20.0.17:35982 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.685771206Z 2024-10-31 11:55:18.685265+00:00 [info] <0.2867396.0> closing AMQP connection <0.2867396.0> (172.20.0.17:36008 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.701179780Z 2024-10-31 11:55:18.700432+00:00 [info] <0.2867390.0> closing AMQP connection <0.2867390.0> (172.20.0.17:35972 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.701495991Z 2024-10-31 11:55:18.701011+00:00 [info] <0.2867400.0> closing AMQP connection <0.2867400.0> (172.20.0.17:36044 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

Ingestion platform logs:

2024-10-31T11:52:45.939468871Z ERR Update indexing fail | category=APP errors=[{"attributes":{"documentId":"cabc3734-22e5-4ace-8111-202e1b6dc6ae","entityType":"","genre":"TECHNICAL","http_status":500},"message":"Update indexing fail","name":"DATABASE_ERROR","stack":"GraphQLError: Update indexing fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:3431:11\n    at async Promise.all (index 0)\n    at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:2136:7)\n    at updateManagerConfigurationLastRun (/opt/opencti/build/src/modules/managerConfiguration/managerConfiguration-domain.ts:67:39)\n    at fileIndexHandler (/opt/opencti/build/src/manager/fileIndexManager.ts:157:11)\n    at /opt/opencti/build/src/manager/fileIndexManager.ts:210:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.UpdateApi [as update] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/update.ts:83:10)\n    at async Promise.all (index 0)\n    at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:2136:7)\n    at updateManagerConfigurationLastRun (/opt/opencti/build/src/modules/managerConfiguration/managerConfiguration-domain.ts:67:39)\n    at fileIndexHandler (/opt/opencti/build/src/manager/fileIndexManager.ts:157:11)\n    at /opt/opencti/build/src/manager/fileIndexManager.ts:210:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}] manager=FILE_INDEX_MANAGER source=backend timestamp=2024-10-31T11:52:45.939Z version=6.3.6

[2024-10-31T11:52:49.421568603Z {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":2,"should":[{"bool":{"minimum_should_match":2,"should":[{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["entity_type.keyword","parent_types.keyword"],"query":"Note"}}]}},{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["rel_granted.*.keyword"],"query":"ce4ccae8-a524-45d6-b06e-1fd6469ab31f"}},{"multi_match":{"fields":["rel_granted.*.keyword"],"query":"bfcf2a7e-863b-48a8-a09c-f1876b03f607"}}]}}]}},{"bool":{"minimum_should_match":1,"should":[{"range":{"updated_at":{"lte":"2024-10-01T11:52:19.116Z"}}}]}}]}}],"must_not":[]}},"size":1500,"sort":[{"updated_at":{"missing":0,"order":"asc"}},{"internal_id.keyword":{"missing":"_last","order":"asc"}},{"standard_id.keyword":"asc"}]},"index":["opencti_stix_domain_objects*","opencti_stix_core_relationships*","opencti_stix_sighting_relationships*","opencti_stix_cyber_observables*"],"track_total_hits":true}},"message":"Fail to execute engine pagination","name":"DATABASE_ERROR","stack":"GraphQLError: Fail to execute engine pagination\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:3285:15\n    at getElementsToDelete (/opt/opencti/build/src/manager/retentionManager.ts:49:14)\n    at executeProcessing (/opt/opencti/build/src/manager/retentionManager.ts:68:18)\n    at Object.retentionHandler [as handler] (/opt/opencti/build/src/manager/retentionManager.ts:109:7)\n    at cronHandler (/opt/opencti/build/src/manager/managerModule.ts:73:11)\n    at /opt/opencti/build/src/manager/managerModule.ts:132:11\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at getElementsToDelete (/opt/opencti/build/src/manager/retentionManager.ts:49:14)\n    at executeProcessing (/opt/opencti/build/src/manager/retentionManager.ts:68:18)\n    at Object.retentionHandler [as handler] (/opt/opencti/build/src/manager/retentionManager.ts:109:7)\n    at cronHandler (/opt/opencti/build/src/manager/managerModule.ts:73:11)\n    at /opt/opencti/build/src/manager/managerModule.ts:132:11\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","manager":"RETENTION_MANAGER","message":"Fail to execute engine pagination","source":"backend","timestamp":"2024-10-31T11:52:49.420Z","version":"6.3.6"}

2024-10-31T11:52:49.460091385Z ERR Reindexing fail from opencti_stix_cyber_observables-000001 to opencti_deleted_objects | category=APP errors=[{"attributes":{"body":{"dest":{"index":"opencti_deleted_objects"},"script":{"source":"ctx._source.remove('fromType'); ctx._source.remove('toType'); ctx._source.remove('spec_version'); ctx._source.remove('representative'); ctx._source.remove('rel_has-reference'); ctx._source.remove('objectOrganization');"},"source":{"index":"opencti_stix_cyber_observables-000001","query":{"ids":{"values":["ec27a375-6c87-46c7-9099-241425dd86d5"]}}}},"genre":"TECHNICAL","http_status":500},"message":"Reindexing fail from opencti_stix_cyber_observables-000001 to opencti_deleted_objects","name":"DATABASE_ERROR","stack":"GraphQLError: Reindexing fail from opencti_stix_cyber_observables-000001 to opencti_deleted_objects\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:3622:11\n    at async Promise.all (index 0)\n    at elDeleteElements (/opt/opencti/build/src/database/engine.js:3661:5)\n    at internalDeleteElementById (/opt/opencti/build/src/database/middleware.js:3213:7)\n    at deleteElementById (/opt/opencti/build/src/database/middleware.js:3235:32)\n    at executeDelete (/opt/opencti/build/src/manager/taskManager.js:240:5)\n    at executeProcessing (/opt/opencti/build/src/manager/taskManager.js:493:13)\n    at taskHandler (/opt/opencti/build/src/manager/taskManager.js:592:22)\n    at /opt/opencti/build/src/manager/taskManager.js:622:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.ReindexApi [as reindex] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/reindex.ts:79:10)\n    at async Promise.all (index 0)\n    at elDeleteElements (/opt/opencti/build/src/database/engine.js:3661:5)\n    at internalDeleteElementById (/opt/opencti/build/src/database/middleware.js:3213:7)\n    at deleteElementById (/opt/opencti/build/src/database/middleware.js:3235:32)\n    at executeDelete (/opt/opencti/build/src/manager/taskManager.js:240:5)\n    at executeProcessing (/opt/opencti/build/src/manager/taskManager.js:493:13)\n    at taskHandler (/opt/opencti/build/src/manager/taskManager.js:592:22)\n    at /opt/opencti/build/src/manager/taskManager.js:622:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}] source=backend timestamp=2024-10-31T11:52:49.459Z version=6.3.6

2024-10-31T11:52:51.178054994Z ERR Engine unhandled rejection | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"reason":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at listAllEntitiesThroughRelations (/opt/opencti/build/src/database/middleware-loader.ts:420:10)"},"message":"Engine unhandled rejection","name":"UNKNOWN_ERROR","stack":"GraphQLError: Engine unhandled rejection\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)\n    at process.<anonymous> (/opt/opencti/build/src/boot.js:61:16)\n    at process.emit (node:events:519:28)\n    at process.emit (/opt/opencti/build/node_modules/source-map-support/source-map-support.js:516:21)\n    at emitUnhandledRejection (node:internal/process/promises:250:13)\n    at throwUnhandledRejectionsMode (node:internal/process/promises:385:19)\n    at processPromiseRejections (node:internal/process/promises:470:17)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)"}] source=backend timestamp=2024-10-31T11:52:51.177Z version=6.3.6

2024-10-31T11:52:51.208704588Z ERR Engine unhandled rejection | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"reason":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at listAllEntitiesThroughRelations (/opt/opencti/build/src/database/middleware-loader.ts:420:10)"},"message":"Engine unhandled rejection","name":"UNKNOWN_ERROR","stack":"GraphQLError: Engine unhandled rejection\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)\n    at process.<anonymous> (/opt/opencti/build/src/boot.js:61:16)\n    at process.emit (node:events:519:28)\n    at process.emit (/opt/opencti/build/node_modules/source-map-support/source-map-support.js:516:21)\n    at emitUnhandledRejection (node:internal/process/promises:250:13)\n    at throwUnhandledRejectionsMode (node:internal/process/promises:385:19)\n    at processPromiseRejections (node:internal/process/promises:470:17)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)"}] source=backend timestamp=2024-10-31T11:52:51.208Z version=6.3.6

2024-10-31T11:52:51.234328659Z ERR [OPENCTI-MODULE] INGESTION - Ingestion handlers cannot be started | category=APP source=backend timestamp=2024-10-31T11:52:51.234Z version=6.3.6

2024-10-31T11:52:51.512673420Z ERR [OPENCTI-MODULE] INGESTION - Error with csv handler C2IntelFeeds - domainC2s-30day-filter-abused | category=APP source=backend timestamp=2024-10-31T11:52:51.512Z version=6.3.6

2024-10-31T11:52:52.560640464Z 1970/01/01 09:15AM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}},{"terms":{"standard_id.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}},{"terms":{"i_aliases_ids.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","84c47805-5b3e-49cf-bb89-f5b66a3f6b5f","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["84c47805-5b3e-49cf-bb89-f5b66a3f6b5f","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[],"name":"ping"}]] size=75 source=backend timestamp=2024-10-31T11:52:52.560Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.8","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"84c47805-5b3e-49cf-bb89-f5b66a3f6b5f","user_metadata":{}} version=6.3.6

2024-10-31T11:53:10.099797755Z 1970/01/01 10:10AM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}},{"terms":{"standard_id.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}},{"terms":{"i_aliases_ids.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","9e63e52f-35e3-4fdd-a379-d0e3420c96f9","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["9e63e52f-35e3-4fdd-a379-d0e3420c96f9","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[],"name":"ping"}]] size=75 source=backend timestamp=2024-10-31T11:53:10.095Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.41","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"9e63e52f-35e3-4fdd-a379-d0e3420c96f9","user_metadata":{}} version=6.3.6

2024-10-31T11:53:11.138543826Z 1970/01/01 09:36AM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}},{"terms":{"standard_id.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}},{"terms":{"i_aliases_ids.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","a6a4f7c4-4b37-4216-a8c3-7992643e37f6","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["a6a4f7c4-4b37-4216-a8c3-7992643e37f6","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[],"name":"ping"}]] size=75 source=backend timestamp=2024-10-31T11:53:11.138Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.54","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"a6a4f7c4-4b37-4216-a8c3-7992643e37f6","user_metadata":{}} version=6.3.6

2024-10-31T11:55:27.030540899Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-10-31T11:55:27.030Z version=6.3.6

2024-10-31T11:55:28.987924407Z 1970/01/02 09:16PM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}},{"terms":{"standard_id.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}},{"terms":{"i_aliases_ids.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","21ac7f54-8c20-4524-800e-c7bb4d56d777","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["21ac7f54-8c20-4524-800e-c7bb4d56d777","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[[{"is_empty":true,"name":"message","type":"Variable"}],[{"is_empty":true,"name":"inError","type":"Variable"}]],"name":"toProcessed"}]] size=183 source=backend timestamp=2024-10-31T11:55:28.982Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.50","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"21ac7f54-8c20-4524-800e-c7bb4d56d777","user_metadata":{}} version=6.3.6

2024-10-31T11:55:33.093421716Z 1970/01/02 05:06PM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}},{"terms":{"standard_id.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}},{"terms":{"i_aliases_ids.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","95e19279-3f1c-4c66-b5cd-c39d074b1051","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["95e19279-3f1c-4c66-b5cd-c39d074b1051","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[[{"is_empty":true,"name":"message","type":"Variable"}],[{"is_empty":true,"name":"inError","type":"Variable"}]],"name":"toProcessed"}]] size=170 source=backend timestamp=2024-10-31T11:55:33.092Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.43","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"95e19279-3f1c-4c66-b5cd-c39d074b1051","user_metadata":{}} version=6.3.6

2024-10-31T11:55:34.464179387Z ERR [REDIS] Failed to extend resource | category=APP manager=FILE_INDEX_MANAGER source=backend timestamp=2024-10-31T11:55:34.463Z version=6.3.6

2024-10-31T11:55:34.464502471Z ERR [REDIS] Failed to extend resource | category=APP manager=HISTORY_MANAGER source=backend timestamp=2024-10-31T11:55:34.464Z version=6.3.6

2024-10-31T11:55:34.464781366Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-10-31T11:55:34.464Z version=6.3.6

2024-10-31T11:55:34.465054014Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-10-31T11:55:34.464Z version=6.3.6

2024-10-31T11:55:34.465368339Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-10-31T11:55:34.465Z version=6.3.6

2024-10-31T11:56:27.044765790Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-10-31T11:56:27.044Z version=6.3.6
richard-julien commented 2 weeks ago

I @dominictory. So your logs show a lot of "TimeoutError: Request timed out" related to elastic. You also have some CONNRESET. For me it could be a sign of:

dominictory commented 2 weeks ago

I @dominictory. So your logs show a lot of "TimeoutError: Request timed out" related to elastic. You also have some CONNRESET. For me it could be a sign of:

  • undersized elasticsearch
  • Stability issue on redis or elastic
  • network issues Difficult to help more

Feels like network issues, but intermittently as bundles do eventually get processed (albeit slowly), then this stops. In RabbitMQ, I see the below showing that workers keep disconnecting from it. Why might this be happening when the server is not under heavy load? For the health checks, might increasing the interval/timeout/retries values help?

2024-11-06T10:24:56.546011051Z 2024-11-06 10:24:56.545169+00:00 [warning] <0.125861.0> closing AMQP connection <0.125861.0> (172.25.0.48:48816 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.546052965Z 2024-11-06 10:24:56.545169+00:00 [warning] <0.125861.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.547771429Z 2024-11-06 10:24:56.545506+00:00 [warning] <0.125915.0> closing AMQP connection <0.125915.0> (172.25.0.48:48824 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.547831218Z 2024-11-06 10:24:56.545506+00:00 [warning] <0.125915.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.547851409Z 2024-11-06 10:24:56.545698+00:00 [warning] <0.125972.0> closing AMQP connection <0.125972.0> (172.25.0.48:48828 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.547868721Z 2024-11-06 10:24:56.545698+00:00 [warning] <0.125972.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.547987644Z 2024-11-06 10:24:56.545764+00:00 [warning] <0.126032.0> closing AMQP connection <0.126032.0> (172.25.0.48:48832 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.548028644Z 2024-11-06 10:24:56.545764+00:00 [warning] <0.126032.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.549025465Z 2024-11-06 10:24:56.546536+00:00 [warning] <0.126245.0> closing AMQP connection <0.126245.0> (172.25.0.48:48862 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.549044703Z 2024-11-06 10:24:56.546536+00:00 [warning] <0.126245.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.549094285Z 2024-11-06 10:24:56.546632+00:00 [warning] <0.126142.0> closing AMQP connection <0.126142.0> (172.25.0.48:48846 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.549116221Z 2024-11-06 10:24:56.546632+00:00 [warning] <0.126142.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.550258484Z 2024-11-06 10:24:56.546747+00:00 [warning] <0.126353.0> closing AMQP connection <0.126353.0> (172.25.0.48:47202 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.550284374Z 2024-11-06 10:24:56.546747+00:00 [warning] <0.126353.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.550574367Z 2024-11-06 10:24:56.546984+00:00 [warning] <0.126498.0> closing AMQP connection <0.126498.0> (172.25.0.48:47214 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.550599060Z 2024-11-06 10:24:56.546984+00:00 [warning] <0.126498.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.550614094Z 2024-11-06 10:24:56.547603+00:00 [warning] <0.126580.0> closing AMQP connection <0.126580.0> (172.25.0.48:47220 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.550627583Z 2024-11-06 10:24:56.547603+00:00 [warning] <0.126580.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.550875141Z 2024-11-06 10:24:56.547523+00:00 [warning] <0.130067.0> closing AMQP connection <0.130067.0> (172.25.0.48:51788 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.550900211Z 2024-11-06 10:24:56.547523+00:00 [warning] <0.130067.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.550915287Z 2024-11-06 10:24:56.547762+00:00 [warning] <0.129946.0> closing AMQP connection <0.129946.0> (172.25.0.48:51758 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.550928862Z 2024-11-06 10:24:56.547762+00:00 [warning] <0.129946.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.550942628Z 2024-11-06 10:24:56.548075+00:00 [warning] <0.130000.0> closing AMQP connection <0.130000.0> (172.25.0.48:51768 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.550982067Z 2024-11-06 10:24:56.548075+00:00 [warning] <0.130000.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.551203591Z 2024-11-06 10:24:56.547898+00:00 [warning] <0.129982.0> closing AMQP connection <0.129982.0> (172.25.0.48:51764 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.551225365Z 2024-11-06 10:24:56.547898+00:00 [warning] <0.129982.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.551239063Z 2024-11-06 10:24:56.548370+00:00 [warning] <0.130025.0> closing AMQP connection <0.130025.0> (172.25.0.48:51770 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.551252206Z 2024-11-06 10:24:56.548370+00:00 [warning] <0.130025.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.551268711Z 2024-11-06 10:24:56.548369+00:00 [warning] <0.130042.0> closing AMQP connection <0.130042.0> (172.25.0.48:51776 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.551283733Z 2024-11-06 10:24:56.548369+00:00 [warning] <0.130042.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.551298110Z 2024-11-06 10:24:56.549036+00:00 [warning] <0.130085.0> closing AMQP connection <0.130085.0> (172.25.0.48:51796 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.551311371Z 2024-11-06 10:24:56.549036+00:00 [warning] <0.130085.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.551750907Z 2024-11-06 10:24:56.549278+00:00 [warning] <0.130102.0> closing AMQP connection <0.130102.0> (172.25.0.48:51804 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.551819033Z 2024-11-06 10:24:56.549278+00:00 [warning] <0.130102.0> client unexpectedly closed TCP connection

2024-11-06T10:24:56.551870926Z 2024-11-06 10:24:56.549286+00:00 [warning] <0.130119.0> closing AMQP connection <0.130119.0> (172.25.0.48:51812 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:56.551887024Z 2024-11-06 10:24:56.549286+00:00 [warning] <0.130119.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.602306974Z 2024-11-06 10:24:57.601362+00:00 [warning] <0.125937.0> closing AMQP connection <0.125937.0> (172.25.0.29:47568 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.602349697Z 2024-11-06 10:24:57.601362+00:00 [warning] <0.125937.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.602359197Z 2024-11-06 10:24:57.601628+00:00 [warning] <0.125898.0> closing AMQP connection <0.125898.0> (172.25.0.29:47566 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.602367122Z 2024-11-06 10:24:57.601628+00:00 [warning] <0.125898.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.602473680Z 2024-11-06 10:24:57.601545+00:00 [warning] <0.125842.0> closing AMQP connection <0.125842.0> (172.25.0.29:47564 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.602506058Z 2024-11-06 10:24:57.601545+00:00 [warning] <0.125842.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.602913590Z 2024-11-06 10:24:57.602551+00:00 [warning] <0.126458.0> closing AMQP connection <0.126458.0> (172.25.0.29:33450 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.602929170Z 2024-11-06 10:24:57.602551+00:00 [warning] <0.126458.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.603039455Z 2024-11-06 10:24:57.602743+00:00 [warning] <0.126010.0> closing AMQP connection <0.126010.0> (172.25.0.29:47578 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.603062545Z 2024-11-06 10:24:57.602743+00:00 [warning] <0.126010.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.603439729Z 2024-11-06 10:24:57.602953+00:00 [warning] <0.126228.0> closing AMQP connection <0.126228.0> (172.25.0.29:47596 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.603462200Z 2024-11-06 10:24:57.602953+00:00 [warning] <0.126228.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.603611859Z 2024-11-06 10:24:57.602910+00:00 [warning] <0.126127.0> closing AMQP connection <0.126127.0> (172.25.0.29:47592 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.603627466Z 2024-11-06 10:24:57.602910+00:00 [warning] <0.126127.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.603849466Z 2024-11-06 10:24:57.603163+00:00 [warning] <0.126328.0> closing AMQP connection <0.126328.0> (172.25.0.29:47612 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.603866117Z 2024-11-06 10:24:57.603163+00:00 [warning] <0.126328.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.604468578Z 2024-11-06 10:24:57.603835+00:00 [warning] <0.126527.0> closing AMQP connection <0.126527.0> (172.25.0.29:33462 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.604486050Z 2024-11-06 10:24:57.603835+00:00 [warning] <0.126527.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605163023Z 2024-11-06 10:24:57.603902+00:00 [warning] <0.129793.0> closing AMQP connection <0.129793.0> (172.25.0.29:41496 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605198165Z 2024-11-06 10:24:57.603902+00:00 [warning] <0.129793.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605215790Z 2024-11-06 10:24:57.603949+00:00 [warning] <0.129810.0> closing AMQP connection <0.129810.0> (172.25.0.29:41510 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605246026Z 2024-11-06 10:24:57.603949+00:00 [warning] <0.129810.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605312953Z 2024-11-06 10:24:57.604216+00:00 [warning] <0.129844.0> closing AMQP connection <0.129844.0> (172.25.0.29:41532 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605338413Z 2024-11-06 10:24:57.604216+00:00 [warning] <0.129844.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605463009Z 2024-11-06 10:24:57.604228+00:00 [warning] <0.129827.0> closing AMQP connection <0.129827.0> (172.25.0.29:41524 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605475769Z 2024-11-06 10:24:57.604228+00:00 [warning] <0.129827.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605535648Z 2024-11-06 10:24:57.604413+00:00 [warning] <0.129861.0> closing AMQP connection <0.129861.0> (172.25.0.29:41534 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605547398Z 2024-11-06 10:24:57.604413+00:00 [warning] <0.129861.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605721591Z 2024-11-06 10:24:57.604569+00:00 [warning] <0.129878.0> closing AMQP connection <0.129878.0> (172.25.0.29:41540 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605734739Z 2024-11-06 10:24:57.604569+00:00 [warning] <0.129878.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605755636Z 2024-11-06 10:24:57.605338+00:00 [warning] <0.129895.0> closing AMQP connection <0.129895.0> (172.25.0.29:41556 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605764098Z 2024-11-06 10:24:57.605338+00:00 [warning] <0.129895.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605890008Z 2024-11-06 10:24:57.605349+00:00 [warning] <0.129913.0> closing AMQP connection <0.129913.0> (172.25.0.29:41560 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605903232Z 2024-11-06 10:24:57.605349+00:00 [warning] <0.129913.0> client unexpectedly closed TCP connection

2024-11-06T10:24:57.605962844Z 2024-11-06 10:24:57.605473+00:00 [warning] <0.129931.0> closing AMQP connection <0.129931.0> (172.25.0.29:41574 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'):

2024-11-06T10:24:57.605988721Z 2024-11-06 10:24:57.605473+00:00 [warning] <0.129931.0> client unexpectedly closed TCP connection

I also see the below RabbitMQ errors:

2024-11-06T10:24:04.761345202Z 2024-11-06 10:24:04.760892+00:00 [error] <0.133857.0> Channel error on connection <0.133847.0> (172.25.0.29:53380 -> 172.25.0.15:5672, vhost: '/', user: 'opencti'), channel 1:

2024-11-06T10:24:04.761378515Z 2024-11-06 10:24:04.760892+00:00 [error] <0.133857.0> operation basic.consume caused a channel exception not_found: no queue 'push_26387a1c-edd1-444b-baa6-a2de3cc3f9a6' in vhost '/'
richard-julien commented 2 weeks ago

Workers directly consume the rabbitmq to process the messages. If you have connection errors here it could be a problem on rabbitmq or a general problem in your network, really hard to know

dominictory commented 3 days ago

Workers directly consume the rabbitmq to process the messages. If you have connection errors here it could be a problem on rabbitmq or a general problem in your network, really hard to know

Hi, I notice when ingestion goes to 0, I see Redis errors. Not sure why as there is plenty resource available. Interestingly, I see this:

2024-11-19T13:51:38.890688789Z ERR Http call interceptor fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500},"message":"Http call interceptor fail","name":"UNKNOWN_ERROR","stack":"GraphQLError: Http call interceptor fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)\n    at fn (/opt/opencti/build/src/http/httpPlatform.js:510:19)\n    at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:71:5)\n    at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n    at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n    at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n    at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n    at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n    at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n    at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n    at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n    at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n    at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n    at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n    at Qge.handle_error (/opt/opencti/build/node_modules/express/lib/router/layer.js:67:12)\n    at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:326:13)\n    at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n    at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)"},{"message":"stream is not readable","name":"InternalServerError","stack":"InternalServerError: stream is not readable\n    at readStream (/opt/opencti/build/node_modules/raw-body/index.js:185:17)\n    at getBody (/opt/opencti/build/node_modules/raw-body/index.js:116:12)\n    at read (/opt/opencti/build/node_modules/body-parser/lib/read.js:79:3)\n    at fn (/opt/opencti/build/node_modules/body-parser/lib/types/json.js:138:5)\n    at Qge.handle [as handle_request] (/opt/opencti/build/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:328:13)\n    at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n    at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n    at fn (/opt/opencti/build/node_modules/passport/lib/middleware/initialize.js:98:5)\n    at Qge.handle [as handle_request] (/opt/opencti/build/node_modules/express/lib/router/layer.js:95:5)\n    at trim_prefix (/opt/opencti/build/node_modules/express/lib/router/index.js:328:13)\n    at done (/opt/opencti/build/node_modules/express/lib/router/index.js:286:9)\n    at Function.process_params (/opt/opencti/build/node_modules/express/lib/router/index.js:346:12)\n    at next (/opt/opencti/build/node_modules/express/lib/router/index.js:280:10)\n    at cb (/opt/opencti/build/node_modules/express-session/index.js:514:7)\n    at cb (/opt/opencti/build/src/database/sessionStore-redis.js:38:14)\n    at done (/opt/opencti/build/node_modules/async-lock/lib/index.js:104:6)\n    at done (/opt/opencti/build/node_modules/async-lock/lib/index.js:157:7)\n    at /opt/opencti/build/src/database/sessionStore-redis.js:34:16\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)"}] source=backend timestamp=2024-11-19T13:51:38.890Z version=6.3.12

Followed by:

2024-11-19T13:52:13.204347378Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T13:52:13.203Z version=6.3.12

2024-11-19T13:52:13.205227759Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T13:52:13.205Z version=6.3.12

2024-11-19T14:04:28.561901304Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:04:28.561Z version=6.3.12

2024-11-19T14:04:28.562351859Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-11-19T14:04:28.562Z version=6.3.12

2024-11-19T14:04:28.704856463Z ERR [REDIS] Failed to extend resource | category=APP manager=SYNC_MANAGER source=backend timestamp=2024-11-19T14:04:28.704Z version=6.3.12

2024-11-19T14:04:56.050114555Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:04:56.049Z version=6.3.12

2024-11-19T14:04:56.050749282Z ERR [REDIS] Failed to extend resource | category=APP manager=FILE_INDEX_MANAGER source=backend timestamp=2024-11-19T14:04:56.050Z version=6.3.12

2024-11-19T14:04:56.051124843Z ERR [REDIS] Failed to extend resource | category=APP manager=HISTORY_MANAGER source=backend timestamp=2024-11-19T14:04:56.050Z version=6.3.12

2024-11-19T14:05:28.668389192Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:05:28.668Z version=6.3.12

2024-11-19T14:05:28.669282576Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:05:28.669Z version=6.3.12

2024-11-19T14:06:28.227151449Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:06:28.227201678Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:06:38.232223781Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:06:41.433610954Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:06:43.783062168Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:31:32.974197302Z ERR [REDIS] Failed to extend resource | category=APP manager=SYNC_MANAGER source=backend timestamp=2024-11-19T14:31:32.973Z version=6.3.12

2024-11-19T14:31:51.087401987Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:31:51.086Z version=6.3.12

2024-11-19T14:31:51.088535129Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-11-19T14:31:51.088Z version=6.3.12

2024-11-19T14:31:51.088981302Z ERR [REDIS] Failed to extend resource | category=APP manager=HISTORY_MANAGER source=backend timestamp=2024-11-19T14:31:51.088Z version=6.3.12

2024-11-19T14:31:51.089486082Z ERR [REDIS] Failed to extend resource | category=APP manager=FILE_INDEX_MANAGER source=backend timestamp=2024-11-19T14:31:51.089Z version=6.3.12

2024-11-19T14:31:51.089820070Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:31:51.089Z version=6.3.12

2024-11-19T14:32:32.820099024Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:32:32.819Z version=6.3.12

2024-11-19T14:32:32.821175144Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:32:32.820Z version=6.3.12

2024-11-19T14:33:32.900546556Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:33:32.900588266Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:33:34.183766645Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:33:34.183827267Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:33:38.152781775Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:52:09.116799274Z ERR [REDIS] Failed to extend resource | category=APP manager=RULE_ENGINE source=backend timestamp=2024-11-19T14:52:09.116Z version=6.3.12

2024-11-19T14:52:09.706540422Z ERR [REDIS] Failed to extend resource | category=APP manager=SYNC_MANAGER source=backend timestamp=2024-11-19T14:52:09.706Z version=6.3.12

2024-11-19T14:52:11.360711244Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-11-19T14:52:11.360Z version=6.3.12

2024-11-19T14:52:11.361108232Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:52:11.360Z version=6.3.12

2024-11-19T14:52:11.361651575Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:52:11.361Z version=6.3.12

2024-11-19T14:52:53.205200384Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-11-19T14:52:53.204Z version=6.3.12

2024-11-19T14:52:53.206582203Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-11-19T14:52:53.206Z version=6.3.12

2024-11-19T14:53:55.608742352Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:53:55.608833311Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

2024-11-19T14:53:57.854654459Z (node:7) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [Socket]. MaxListeners is 10. Use emitter.setMaxListeners() to increase limit

Are the Redis errors and MaxListenersExceededWarning logs potentially linked? Is the Redis error somehow due to resource, even though there is plenty of headroom available on the server? The case is always the same, I restart the platform, everything comes up as normal, ingestion is fast (20-30 bundles/sec), then after some time passes, it slows down, eventually going to 0.

I'm no expert, but I came across the below that talks about MaxListenersExceededWarning in the context of Redis client:

https://stackoverflow.com/questions/50709059/maxlistenersexceededwarning-possible-eventemitter-memory-leak-detected-11-mess

dominictory commented 3 days ago

I say this as since upgrading to 6.3.12, I again removed all connectors in UI and reset RabbitMQ. I see no more RabbitMQ errors/warnings since doing so. The below is something I haven't seen before with Redis, is it concering? It appears multiple times.

1:M 19 Nov 2024 14:50:54.124 # Client id=10620 addr=172.27.0.10:39176 laddr=172.27.0.39:6379 fd=13 name= age=34 idle=1 flags=P db=0 sub=0 psub=2 ssub=0 multi=-1 watch=0 qbuf=0 qbuf-free=20474 argv-mem=0 multi-mem=0 rbs=1024 rbp=0 obl=0 oll=3 omem=34078792 tot-mem=34101272 events=rw cmd=psubscribe user=default redir=-1 resp=2 lib-name= lib-ver= scheduled to be closed ASAP for overcoming of output buffer limits.

Similar message on https://github.com/OpenCTI-Platform/opencti/issues/4936

dominictory commented 2 days ago

There's these HUGE logs in worker logs too. This one in particular was due to AlienVault connector which currently has ~2k bundles queued and not getting processed:

worker.log

This was followed by the above Redis errors.