No ingestion, low resource usage, connections reset

[dominictory]

Description

At present, I have 68k queued bundles, of which <1 are getting processed/sec, as below, which is constant at the moment. In worker and connector logs, whilst some bundles do get processed, there are sporadic connection reset errors, which point to RabbitMQ which is showing as healthy in the stack, as below, with no errors other than the occasional errors below every 5 mins or so. The backend platform does show as unhealthy, with numerous errors, as below at the bottom. Connectors seem to go between active and inactive regularly. I would appreciate some help troubleshooting further please, and especially with Redis/platform if I can try any different configurations. Thanks :)

2024-10-31T11:54:23.631945580Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> closing AMQP connection <0.2750204.0> (172.20.0.43:51702 -> 172.20.0.9:5672):

2024-10-31T11:54:23.632025790Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> missed heartbeats from client, timeout: 10s

My docker-compose setup is as follows:

1x frontend platform (users, internal export/import connectors only, no managers) 1x backend platform (3x workers, connectors, all managers)

VM: Ubuntu 22.04 LTS; 16 cores; 64GB RAM

Current load average: 7.22, 5.52, 4.45 Memory usage: total used free shared buff/cache available Mem: 62Gi 48Gi 4.1Gi 43Mi 10Gi 13Gi Swap: 8.0Gi 5.4Gi 2.6Gi

docker-compose.yml sample:

services:
  redis:
    image: redis:${REDIS_VERSION}
    restart: always
    volumes:
      - redisdata3:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 5s
      retries: 3
  elasticsearch:
    image: elasticsearch:${ELK_VERSION}
    volumes:
      - esdata:/usr/share/elasticsearch/data
    environment:
      - discovery.type=single-node
      - xpack.ml.enabled=false
      - xpack.security.enabled=false
      - thread_pool.search.queue_size=5000
      - logger.org.elasticsearch.discovery="ERROR"
      - "ES_JAVA_OPTS=-Xms${ELASTIC_MEMORY_SIZE} -Xmx${ELASTIC_MEMORY_SIZE}"
    restart: always
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536
        hard: 65536
    healthcheck:
      test: curl -s http://elasticsearch:9200 >/dev/null || exit 1
      interval: 30s
      timeout: 10s
      retries: 50
  minio:
    image: minio/minio:${MINIO_VERSION}
    volumes:
      - s3data:/data
    ports:
      - "9000:9000"
    environment:
      MINIO_ROOT_USER: ${MINIO_ROOT_USER}
      MINIO_ROOT_PASSWORD: ${MINIO_ROOT_PASSWORD}
    command: server /data
    restart: always
    healthcheck:
      test: ["CMD", "mc", "ready", "local"]
      interval: 10s
      timeout: 5s
      retries: 3
  rabbitmq:
    image: rabbitmq:${RMQ_VERSION}
    environment:
      - RABBITMQ_DEFAULT_USER=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ_DEFAULT_PASS=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ__CONSUMER_TIMEOUT=36000000
    volumes:
      - amqpdata:/var/lib/rabbitmq
    restart: always
    healthcheck:
      test: rabbitmq-diagnostics -q ping
      interval: 30s
      timeout: 30s
      retries: 3
  opencti-front:
    image: opencti/platform:${OPENCTI_VERSION}
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=443
      - APP__BASE_URL=https://localhost
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - APP__APP_LOGS__LOGS_FILES=false
      - APP__SESSION_TIMEOUT=28800000
      - APP__HTTPS_CERT__KEY=/certs/cti.key
      - APP__HTTPS_CERT__CRT=/certs/cti.crt
      - RULE_ENGINE__ENABLED=false
      - HISTORY_MANAGER__ENABLED=false
      - TASK_SCHEDULER__ENABLED=false
      - SYNC_MANAGER__ENABLED=false
      - EXPIRATION_SCHEDULER__ENABLED=false
      - RETENTION_MANAGER__ENABLED=false
      - NOTIFICATION_MANAGER__ENABLED=false
      - PUBLISHER_MANAGER__ENABLED=false
      - INGESTION_MANAGER__ENABLED=false
      - PLAYBOOK_MANAGER__ENABLED=false
      - ACTIVITY_MANAGER__ENABLED=false
      - CONNECTOR_MANAGER__ENABLED=false
      - IMPORT_CSV_BUILT_IN_CONNECTOR__ENABLED=false
      - FILE_INDEX_MANAGER__ENABLED=false
      - INDICATOR_DECAY_MANAGER__ENABLED=false
      - GARBAGE_COLLECTION_MANAGER__ENABLED=false
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ__QUEUE_TYPE="quorum"
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
      - APP__HEALTH_ACCESS_KEY=${OPENCTI_HEALTHCHECK_ACCESS_KEY}
      #- XTM__OPENBAS_URL=http://10.254.203.11:8080
      #- XTM__OPENBAS_TOKEN=803889f7-c97d-4899-b984-4f575297d771
    volumes:
      - tls:/certs
    ports:
      - "443:443"
    restart: always
  opencti-data:
    image: opencti/platform:${OPENCTI_VERSION}
    environment:
      - NODE_OPTIONS=--max-old-space-size=8096
      - APP__PORT=8080
      - APP__BASE_URL=http://localhost:8080
      - APP__ADMIN__EMAIL=${OPENCTI_ADMIN_EMAIL}
      - APP__ADMIN__PASSWORD=${OPENCTI_ADMIN_PASSWORD}
      - APP__ADMIN__TOKEN=${OPENCTI_ADMIN_TOKEN}
      - APP__APP_LOGS__LOGS_LEVEL=error
      - APP__APP_LOGS__LOGS_FILES=false
      - RULE_ENGINE__ENABLED=false
      - REDIS__HOSTNAME=redis
      - REDIS__PORT=6379
      - ELASTICSEARCH__URL=http://elasticsearch:9200
      - MINIO__ENDPOINT=minio
      - MINIO__PORT=9000
      - MINIO__USE_SSL=false
      - MINIO__ACCESS_KEY=${MINIO_ROOT_USER}
      - MINIO__SECRET_KEY=${MINIO_ROOT_PASSWORD}
      - RABBITMQ__HOSTNAME=rabbitmq
      - RABBITMQ__USERNAME=${RABBITMQ_DEFAULT_USER}
      - RABBITMQ__PASSWORD=${RABBITMQ_DEFAULT_PASS}
      - RABBITMQ__QUEUE_TYPE="quorum"
      - PROVIDERS__LOCAL__STRATEGY=LocalStrategy
      - APP__HEALTH_ACCESS_KEY=${OPENCTI_HEALTHCHECK_ACCESS_KEY}
    depends_on:
      redis:
        condition: service_healthy
      elasticsearch:
        condition: service_healthy
      minio:
        condition: service_healthy
      rabbitmq:
        condition: service_healthy
    ports:
      - "8080:8080"
    restart: always
    healthcheck:
      test:  ["CMD", "wget", "-qO-", "http://opencti-data:8080/health?health_access_key=${OPENCTI_HEALTHCHECK_ACCESS_KEY}"]
      interval: 10s
      timeout: 5s
      retries: 100
  worker:
    image: opencti/worker:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=http://opencti-data:8080
      - OPENCTI_TOKEN=${OPENCTI_ADMIN_TOKEN}
      - WORKER_LOG_LEVEL=error
    depends_on:
      opencti-data:
        condition: service_healthy
    deploy:
      mode: replicated
      replicas: 3
    restart: always
  connector-export-file-stix:
    image: opencti/connector-export-file-stix:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=https://opencti-front
      - OPENCTI_TOKEN=ebf85396-544e-4a78-ab76-b66c5ea0b663
      - CONNECTOR_ID=dd817c8b-abae-460a-9ebc-97b1551e70e6
      - CONNECTOR_NAME=ExportFileStix2
      - CONNECTOR_SCOPE=application/vnd.oasis.stix+json
      - CONNECTOR_LOG_LEVEL=error
    restart: always
  connector-alienvault:
    image: opencti/connector-alienvault:${OPENCTI_VERSION}
    environment:
      - OPENCTI_URL=http://opencti-data:8080
      - OPENCTI_TOKEN=bd3bb843-cafe-451d-a5a7-711d32494ddb
      - CONNECTOR_ID=c1af30c7-b667-4167-9fcb-693972e61091
      - CONNECTOR_NAME=AlienVault
      - CONNECTOR_SCOPE=alienvault
      - CONNECTOR_LOG_LEVEL=error
      - CONNECTOR_DURATION_PERIOD=PT30M
      - ALIENVAULT_BASE_URL=https://otx.alienvault.com
      - ALIENVAULT_API_KEY=redacted
      - ALIENVAULT_TLP=White
      - ALIENVAULT_CREATE_OBSERVABLES=true
      - ALIENVAULT_CREATE_INDICATORS=true
      - ALIENVAULT_PULSE_START_TIMESTAMP=2024-01-01T00:00:00
      - ALIENVAULT_REPORT_TYPE=threat-report
      - ALIENVAULT_REPORT_STATUS=New
      - ALIENVAULT_GUESS_MALWARE=true
      - ALIENVAULT_GUESS_CVE=true
      - ALIENVAULT_EXCLUDED_PULSE_INDICATOR_TYPES=FileHash-MD5,FileHash-SHA1
      - ALIENVAULT_ENABLE_RELATIONSHIPS=true
      - ALIENVAULT_ENABLE_ATTACK_PATTERNS_INDICATES=true
    restart: always
...
volumes:
  esdata:
    external: true
    name: opencti_esdata
  s3data:
    external: true
    name: opencti_s3data
  redisdata3:
    external: true
    name: opencti_redisdata3
  amqpdata:
    external: true
    name: opencti_amqpdata
  tls:
    external: true
    name: opencti_tls

.env:

OPENCTI_ADMIN_EMAIL=admin@opencti.ct
OPENCTI_ADMIN_PASSWORD=@;0.Sx>{G5).Mzn#^VvU
OPENCTI_ADMIN_TOKEN=4e95798b-21a6-4571-819f-d974f2c2abea
OPENCTI_VERSION=6.3.6
REDIS_VERSION=7.4.0
ELK_VERSION=8.15.2
ELASTIC_MEMORY_SIZE=32G
MINIO_VERSION=RELEASE.2024-05-28T17-19-04Z
RMQ_VERSION=3.13-management
MINIO_ROOT_USER=opencti
MINIO_ROOT_PASSWORD=OMYPQ5skRD8WU6v^iIRL
RABBITMQ_DEFAULT_USER=opencti
RABBITMQ_DEFAULT_PASS=T+9ix{l{9>U<i?KZ>@B#
OPENCTI_HEALTHCHECK_ACCESS_KEY=dccc470f-7339-4026-a7cd-2ab6c82a480c

RabbitMQ log sample:

2024-10-31T11:53:21.165303199Z 2024-10-31 11:53:21.164784+00:00 [info] <0.2864758.0> accepting AMQP connection <0.2864758.0> (172.20.0.54:35830 -> 172.20.0.9:5672)

2024-10-31T11:53:21.168424077Z 2024-10-31 11:53:21.168055+00:00 [info] <0.2864758.0> connection <0.2864758.0> (172.20.0.54:35830 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:54:23.631945580Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> closing AMQP connection <0.2750204.0> (172.20.0.43:51702 -> 172.20.0.9:5672):

2024-10-31T11:54:23.632025790Z 2024-10-31 11:54:23.631087+00:00 [error] <0.2750204.0> missed heartbeats from client, timeout: 10s

2024-10-31T11:55:18.580620899Z 2024-10-31 11:55:18.580178+00:00 [info] <0.2867390.0> accepting AMQP connection <0.2867390.0> (172.20.0.17:35972 -> 172.20.0.9:5672)

2024-10-31T11:55:18.580770860Z 2024-10-31 11:55:18.580435+00:00 [info] <0.2867388.0> accepting AMQP connection <0.2867388.0> (172.20.0.17:35982 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581007160Z 2024-10-31 11:55:18.580566+00:00 [info] <0.2867380.0> accepting AMQP connection <0.2867380.0> (172.20.0.17:35992 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581423563Z 2024-10-31 11:55:18.580647+00:00 [info] <0.2867396.0> accepting AMQP connection <0.2867396.0> (172.20.0.17:36008 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581579732Z 2024-10-31 11:55:18.580829+00:00 [info] <0.2867384.0> accepting AMQP connection <0.2867384.0> (172.20.0.17:36018 -> 172.20.0.9:5672)

2024-10-31T11:55:18.581645892Z 2024-10-31 11:55:18.581004+00:00 [info] <0.2867381.0> accepting AMQP connection <0.2867381.0> (172.20.0.17:36034 -> 172.20.0.9:5672)

2024-10-31T11:55:18.582195331Z 2024-10-31 11:55:18.581093+00:00 [info] <0.2867400.0> accepting AMQP connection <0.2867400.0> (172.20.0.17:36044 -> 172.20.0.9:5672)

2024-10-31T11:55:18.582210797Z 2024-10-31 11:55:18.581270+00:00 [info] <0.2867402.0> accepting AMQP connection <0.2867402.0> (172.20.0.17:36056 -> 172.20.0.9:5672)

2024-10-31T11:55:18.628130363Z 2024-10-31 11:55:18.627511+00:00 [info] <0.2867380.0> connection <0.2867380.0> (172.20.0.17:35992 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628178378Z 2024-10-31 11:55:18.627577+00:00 [info] <0.2867402.0> connection <0.2867402.0> (172.20.0.17:36056 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628237012Z 2024-10-31 11:55:18.627640+00:00 [info] <0.2867381.0> connection <0.2867381.0> (172.20.0.17:36034 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628426966Z 2024-10-31 11:55:18.627773+00:00 [info] <0.2867384.0> connection <0.2867384.0> (172.20.0.17:36018 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628548824Z 2024-10-31 11:55:18.627652+00:00 [info] <0.2867396.0> connection <0.2867396.0> (172.20.0.17:36008 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.628614181Z 2024-10-31 11:55:18.627894+00:00 [info] <0.2867388.0> connection <0.2867388.0> (172.20.0.17:35982 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.635481580Z 2024-10-31 11:55:18.635034+00:00 [info] <0.2867400.0> connection <0.2867400.0> (172.20.0.17:36044 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.635547523Z 2024-10-31 11:55:18.635117+00:00 [info] <0.2867390.0> connection <0.2867390.0> (172.20.0.17:35972 -> 172.20.0.9:5672): user 'opencti' authenticated and granted access to vhost '/'

2024-10-31T11:55:18.658688709Z 2024-10-31 11:55:18.657703+00:00 [info] <0.2867402.0> closing AMQP connection <0.2867402.0> (172.20.0.17:36056 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.673803618Z 2024-10-31 11:55:18.673165+00:00 [info] <0.2867381.0> closing AMQP connection <0.2867381.0> (172.20.0.17:36034 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.673849465Z 2024-10-31 11:55:18.673174+00:00 [info] <0.2867380.0> closing AMQP connection <0.2867380.0> (172.20.0.17:35992 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.685619113Z 2024-10-31 11:55:18.684949+00:00 [info] <0.2867384.0> closing AMQP connection <0.2867384.0> (172.20.0.17:36018 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.685677895Z 2024-10-31 11:55:18.685143+00:00 [info] <0.2867388.0> closing AMQP connection <0.2867388.0> (172.20.0.17:35982 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.685771206Z 2024-10-31 11:55:18.685265+00:00 [info] <0.2867396.0> closing AMQP connection <0.2867396.0> (172.20.0.17:36008 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.701179780Z 2024-10-31 11:55:18.700432+00:00 [info] <0.2867390.0> closing AMQP connection <0.2867390.0> (172.20.0.17:35972 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

2024-10-31T11:55:18.701495991Z 2024-10-31 11:55:18.701011+00:00 [info] <0.2867400.0> closing AMQP connection <0.2867400.0> (172.20.0.17:36044 -> 172.20.0.9:5672, vhost: '/', user: 'opencti')

Ingestion platform logs:

2024-10-31T11:52:45.939468871Z ERR Update indexing fail | category=APP errors=[{"attributes":{"documentId":"cabc3734-22e5-4ace-8111-202e1b6dc6ae","entityType":"","genre":"TECHNICAL","http_status":500},"message":"Update indexing fail","name":"DATABASE_ERROR","stack":"GraphQLError: Update indexing fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:3431:11\n    at async Promise.all (index 0)\n    at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:2136:7)\n    at updateManagerConfigurationLastRun (/opt/opencti/build/src/modules/managerConfiguration/managerConfiguration-domain.ts:67:39)\n    at fileIndexHandler (/opt/opencti/build/src/manager/fileIndexManager.ts:157:11)\n    at /opt/opencti/build/src/manager/fileIndexManager.ts:210:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.UpdateApi [as update] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/update.ts:83:10)\n    at async Promise.all (index 0)\n    at updateAttributeMetaResolved (/opt/opencti/build/src/database/middleware.js:2136:7)\n    at updateManagerConfigurationLastRun (/opt/opencti/build/src/modules/managerConfiguration/managerConfiguration-domain.ts:67:39)\n    at fileIndexHandler (/opt/opencti/build/src/manager/fileIndexManager.ts:157:11)\n    at /opt/opencti/build/src/manager/fileIndexManager.ts:210:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}] manager=FILE_INDEX_MANAGER source=backend timestamp=2024-10-31T11:52:45.939Z version=6.3.6

[2024-10-31T11:52:49.421568603Z {"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":2,"should":[{"bool":{"minimum_should_match":2,"should":[{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["entity_type.keyword","parent_types.keyword"],"query":"Note"}}]}},{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["rel_granted.*.keyword"],"query":"ce4ccae8-a524-45d6-b06e-1fd6469ab31f"}},{"multi_match":{"fields":["rel_granted.*.keyword"],"query":"bfcf2a7e-863b-48a8-a09c-f1876b03f607"}}]}}]}},{"bool":{"minimum_should_match":1,"should":[{"range":{"updated_at":{"lte":"2024-10-01T11:52:19.116Z"}}}]}}]}}],"must_not":[]}},"size":1500,"sort":[{"updated_at":{"missing":0,"order":"asc"}},{"internal_id.keyword":{"missing":"_last","order":"asc"}},{"standard_id.keyword":"asc"}]},"index":["opencti_stix_domain_objects*","opencti_stix_core_relationships*","opencti_stix_sighting_relationships*","opencti_stix_cyber_observables*"],"track_total_hits":true}},"message":"Fail to execute engine pagination","name":"DATABASE_ERROR","stack":"GraphQLError: Fail to execute engine pagination\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:3285:15\n    at getElementsToDelete (/opt/opencti/build/src/manager/retentionManager.ts:49:14)\n    at executeProcessing (/opt/opencti/build/src/manager/retentionManager.ts:68:18)\n    at Object.retentionHandler [as handler] (/opt/opencti/build/src/manager/retentionManager.ts:109:7)\n    at cronHandler (/opt/opencti/build/src/manager/managerModule.ts:73:11)\n    at /opt/opencti/build/src/manager/managerModule.ts:132:11\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at getElementsToDelete (/opt/opencti/build/src/manager/retentionManager.ts:49:14)\n    at executeProcessing (/opt/opencti/build/src/manager/retentionManager.ts:68:18)\n    at Object.retentionHandler [as handler] (/opt/opencti/build/src/manager/retentionManager.ts:109:7)\n    at cronHandler (/opt/opencti/build/src/manager/managerModule.ts:73:11)\n    at /opt/opencti/build/src/manager/managerModule.ts:132:11\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}],"level":"error","manager":"RETENTION_MANAGER","message":"Fail to execute engine pagination","source":"backend","timestamp":"2024-10-31T11:52:49.420Z","version":"6.3.6"}

2024-10-31T11:52:49.460091385Z ERR Reindexing fail from opencti_stix_cyber_observables-000001 to opencti_deleted_objects | category=APP errors=[{"attributes":{"body":{"dest":{"index":"opencti_deleted_objects"},"script":{"source":"ctx._source.remove('fromType'); ctx._source.remove('toType'); ctx._source.remove('spec_version'); ctx._source.remove('representative'); ctx._source.remove('rel_has-reference'); ctx._source.remove('objectOrganization');"},"source":{"index":"opencti_stix_cyber_observables-000001","query":{"ids":{"values":["ec27a375-6c87-46c7-9099-241425dd86d5"]}}}},"genre":"TECHNICAL","http_status":500},"message":"Reindexing fail from opencti_stix_cyber_observables-000001 to opencti_deleted_objects","name":"DATABASE_ERROR","stack":"GraphQLError: Reindexing fail from opencti_stix_cyber_observables-000001 to opencti_deleted_objects\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:3622:11\n    at async Promise.all (index 0)\n    at elDeleteElements (/opt/opencti/build/src/database/engine.js:3661:5)\n    at internalDeleteElementById (/opt/opencti/build/src/database/middleware.js:3213:7)\n    at deleteElementById (/opt/opencti/build/src/database/middleware.js:3235:32)\n    at executeDelete (/opt/opencti/build/src/manager/taskManager.js:240:5)\n    at executeProcessing (/opt/opencti/build/src/manager/taskManager.js:493:13)\n    at taskHandler (/opt/opencti/build/src/manager/taskManager.js:592:22)\n    at /opt/opencti/build/src/manager/taskManager.js:622:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.ReindexApi [as reindex] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/reindex.ts:79:10)\n    at async Promise.all (index 0)\n    at elDeleteElements (/opt/opencti/build/src/database/engine.js:3661:5)\n    at internalDeleteElementById (/opt/opencti/build/src/database/middleware.js:3213:7)\n    at deleteElementById (/opt/opencti/build/src/database/middleware.js:3235:32)\n    at executeDelete (/opt/opencti/build/src/manager/taskManager.js:240:5)\n    at executeProcessing (/opt/opencti/build/src/manager/taskManager.js:493:13)\n    at taskHandler (/opt/opencti/build/src/manager/taskManager.js:592:22)\n    at /opt/opencti/build/src/manager/taskManager.js:622:9\n    at iit.#runHandlerAndScheduleTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:36:13)\n    at Timeout._onTimeout (/opt/opencti/build/node_modules/set-interval-async/dist/set-interval-async-timer.cjs:29:13)"}] source=backend timestamp=2024-10-31T11:52:49.459Z version=6.3.6

2024-10-31T11:52:51.178054994Z ERR Engine unhandled rejection | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"reason":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at listAllEntitiesThroughRelations (/opt/opencti/build/src/database/middleware-loader.ts:420:10)"},"message":"Engine unhandled rejection","name":"UNKNOWN_ERROR","stack":"GraphQLError: Engine unhandled rejection\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)\n    at process.<anonymous> (/opt/opencti/build/src/boot.js:61:16)\n    at process.emit (node:events:519:28)\n    at process.emit (/opt/opencti/build/node_modules/source-map-support/source-map-support.js:516:21)\n    at emitUnhandledRejection (node:internal/process/promises:250:13)\n    at throwUnhandledRejectionsMode (node:internal/process/promises:385:19)\n    at processPromiseRejections (node:internal/process/promises:470:17)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)"}] source=backend timestamp=2024-10-31T11:52:51.177Z version=6.3.6

2024-10-31T11:52:51.208704588Z ERR Engine unhandled rejection | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"reason":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at listAllEntitiesThroughRelations (/opt/opencti/build/src/database/middleware-loader.ts:420:10)"},"message":"Engine unhandled rejection","name":"UNKNOWN_ERROR","stack":"GraphQLError: Engine unhandled rejection\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at UnknownError (/opt/opencti/build/src/config/errors.js:81:47)\n    at process.<anonymous> (/opt/opencti/build/src/boot.js:61:16)\n    at process.emit (node:events:519:28)\n    at process.emit (/opt/opencti/build/node_modules/source-map-support/source-map-support.js:516:21)\n    at emitUnhandledRejection (node:internal/process/promises:250:13)\n    at throwUnhandledRejectionsMode (node:internal/process/promises:385:19)\n    at processPromiseRejections (node:internal/process/promises:470:17)\n    at processTicksAndRejections (node:internal/process/task_queues:96:32)"}] source=backend timestamp=2024-10-31T11:52:51.208Z version=6.3.6

2024-10-31T11:52:51.234328659Z ERR [OPENCTI-MODULE] INGESTION - Ingestion handlers cannot be started | category=APP source=backend timestamp=2024-10-31T11:52:51.234Z version=6.3.6

2024-10-31T11:52:51.512673420Z ERR [OPENCTI-MODULE] INGESTION - Error with csv handler C2IntelFeeds - domainC2s-30day-filter-abused | category=APP source=backend timestamp=2024-10-31T11:52:51.512Z version=6.3.6

2024-10-31T11:52:52.560640464Z 1970/01/01 09:15AM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}},{"terms":{"standard_id.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}},{"terms":{"i_aliases_ids.keyword":["work_a2695e04-6d3d-4496-8ba1-7ed9e13779ea_2024-10-31T11:42:10.327Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","84c47805-5b3e-49cf-bb89-f5b66a3f6b5f","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["84c47805-5b3e-49cf-bb89-f5b66a3f6b5f","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[],"name":"ping"}]] size=75 source=backend timestamp=2024-10-31T11:52:52.560Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.8","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"84c47805-5b3e-49cf-bb89-f5b66a3f6b5f","user_metadata":{}} version=6.3.6

2024-10-31T11:53:10.099797755Z 1970/01/01 10:10AM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}},{"terms":{"standard_id.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}},{"terms":{"i_aliases_ids.keyword":["work_1cb07a10-c686-4113-8701-17eea1977015_2024-10-31T11:27:40.390Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","9e63e52f-35e3-4fdd-a379-d0e3420c96f9","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["9e63e52f-35e3-4fdd-a379-d0e3420c96f9","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[],"name":"ping"}]] size=75 source=backend timestamp=2024-10-31T11:53:10.095Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.41","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"9e63e52f-35e3-4fdd-a379-d0e3420c96f9","user_metadata":{}} version=6.3.6

2024-10-31T11:53:11.138543826Z 1970/01/01 09:36AM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}},{"terms":{"standard_id.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}},{"terms":{"i_aliases_ids.keyword":["work_2a53a279-b03e-4d6c-bcde-eb878d66874a_2024-10-31T11:32:28.198Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","a6a4f7c4-4b37-4216-a8c3-7992643e37f6","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["a6a4f7c4-4b37-4216-a8c3-7992643e37f6","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at pingWork (/opt/opencti/build/src/domain/work.js:110:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[],"name":"ping"}]] size=75 source=backend timestamp=2024-10-31T11:53:11.138Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.54","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"a6a4f7c4-4b37-4216-a8c3-7992643e37f6","user_metadata":{}} version=6.3.6

2024-10-31T11:55:27.030540899Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-10-31T11:55:27.030Z version=6.3.6

2024-10-31T11:55:28.987924407Z 1970/01/02 09:16PM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}},{"terms":{"standard_id.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}},{"terms":{"i_aliases_ids.keyword":["work_22a565c5-6aa4-4064-88d5-c17ab711c994_2024-10-31T11:29:31.533Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","21ac7f54-8c20-4524-800e-c7bb4d56d777","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["21ac7f54-8c20-4524-800e-c7bb4d56d777","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[[{"is_empty":true,"name":"message","type":"Variable"}],[{"is_empty":true,"name":"inError","type":"Variable"}]],"name":"toProcessed"}]] size=183 source=backend timestamp=2024-10-31T11:55:28.982Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.50","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"21ac7f54-8c20-4524-800e-c7bb4d56d777","user_metadata":{}} version=6.3.6

2024-10-31T11:55:33.093421716Z 1970/01/02 05:06PM ERR Find direct ids fail | category=APP errors=[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"terms":{"internal_id.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}},{"terms":{"standard_id.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}},{"terms":{"x_opencti_stix_ids.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}},{"terms":{"i_aliases_ids.keyword":["work_11b78342-5026-453e-8c2f-d6209eb0ce28_2024-10-31T11:40:03.142Z"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"terms":{"entity_type.keyword":["work"]}},{"terms":{"parent_types.keyword":["work"]}}]}},{"bool":{"should":[{"terms":{"authorized_members.id.keyword":["ALL","95e19279-3f1c-4c66-b5cd-c39d074b1051","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}},{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_authorities.keyword":["95e19279-3f1c-4c66-b5cd-c39d074b1051","8763664e-d84b-4ed1-942d-54891bbba7a8","0f8e5d31-56b6-4bda-ad97-af81f28c3283","65aa1b40-2eae-4576-83b2-5e45afd6e2ab","KNOWLEDGE_KNUPDATE_KNDELETE","KNOWLEDGE_KNUPLOAD","KNOWLEDGE_KNASKIMPORT","KNOWLEDGE_KNGETEXPORT_KNASKEXPORT","KNOWLEDGE_KNENRICHMENT","MODULES_MODMANAGE","TAXIIAPI","SETTINGS_SETMARKINGS","SETTINGS_SETLABELS","CONNECTORAPI","KNOWLEDGE_KNUPDATE_KNBYPASSREFERENCE","SETTINGS_SECURITYACTIVITY","INGESTION","INGESTION_SETINGESTIONS","SETTINGS_SETPARAMETERS","SETTINGS_SETCUSTOMIZATION","SETTINGS_FILEINDEXING","SETTINGS_SUPPORT","KNOWLEDGE_KNUPDATE_KNBYPASSFIELDS","65aa1b40-2eae-4576-83b2-5e45afd6e2ab"]}}]}},{"bool":{"minimum_should_match":1,"should":[{"bool":{"must_not":[{"exists":{"field":"rel_object-marking.internal_id.keyword"}}]}},{"bool":{"must_not":[]}}]}}],"must_not":[]}},"sort":[{"created_at":"asc"}]},"index":"opencti_history*","size":5000}},"message":"Find direct ids fail","name":"DATABASE_ERROR","stack":"GraphQLError: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:7:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:57:48)\n    at /opt/opencti/build/src/database/engine.js:1574:15\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"},{"message":"Request timed out","name":"TimeoutError","stack":"TimeoutError: Request timed out\n    at q7t._request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:630:36)\n    at /opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:704:22\n    at q7t.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:701:14)\n    at $$e.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:96:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1573:20)\n    at elLoadById (/opt/opencti/build/src/database/engine.js:1599:16)\n    at loadWorkById (/opt/opencti/build/src/domain/work.js:34:18)\n    at updateProcessedTime (/opt/opencti/build/src/domain/work.js:251:23)"}] inner_relation_creation=0 operation=Unspecified query_attributes=[[{"arguments":[[{"is_empty":true,"name":"message","type":"Variable"}],[{"is_empty":true,"name":"inError","type":"Variable"}]],"name":"toProcessed"}]] size=170 source=backend timestamp=2024-10-31T11:55:33.092Z type=WRITE_ERROR user={"group_ids":["0f8e5d31-56b6-4bda-ad97-af81f28c3283"],"ip":"::ffff:172.20.0.43","organization_ids":["8763664e-d84b-4ed1-942d-54891bbba7a8"],"socket":"query","user_id":"95e19279-3f1c-4c66-b5cd-c39d074b1051","user_metadata":{}} version=6.3.6

2024-10-31T11:55:34.464179387Z ERR [REDIS] Failed to extend resource | category=APP manager=FILE_INDEX_MANAGER source=backend timestamp=2024-10-31T11:55:34.463Z version=6.3.6

2024-10-31T11:55:34.464502471Z ERR [REDIS] Failed to extend resource | category=APP manager=HISTORY_MANAGER source=backend timestamp=2024-10-31T11:55:34.464Z version=6.3.6

2024-10-31T11:55:34.464781366Z ERR [REDIS] Failed to extend resource | category=APP manager=PUBLISHER_MANAGER source=backend timestamp=2024-10-31T11:55:34.464Z version=6.3.6

2024-10-31T11:55:34.465054014Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-10-31T11:55:34.464Z version=6.3.6

2024-10-31T11:55:34.465368339Z ERR [REDIS] Failed to extend resource | category=APP manager=NOTIFICATION_MANAGER source=backend timestamp=2024-10-31T11:55:34.465Z version=6.3.6

2024-10-31T11:56:27.044765790Z ERR [REDIS] Failed to extend resource | category=APP manager=PLAYBOOK_MANAGER source=backend timestamp=2024-10-31T11:56:27.044Z version=6.3.6

[richard-julien]

I @dominictory. So your logs show a lot of "TimeoutError: Request timed out" related to elastic. You also have some CONNRESET. For me it could be a sign of:

• undersized elasticsearch
• Stability issue on redis or elastic
• network issues Difficult to help more