search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.08k stars 903 forks source link

Mandiant Connector crashes Elasticsearch DB with malformed STIX IOCs #2327

Open faustus25 opened 2 years ago

faustus25 commented 2 years ago

Prerequisites

No issue specific to this reported to this Connector.

Description

Manual install of OpenCTI (environment info)

Having ran the Mandiant connector to download content from the last 90 days the worker task never completes due to certain IOCs rendering in malformed STIX which crashes the Elasticsearch DB and requires a restart on the Elasticsearch service (sometimes forcing a service restart) and re-running OpenCTI.
The problem is start I cannot run any other connectors until the import finishes so I need to know how to stop the import. Other connectors have ran fine without causing this issue.

The only workaround is too keep restarting the Elasticsearch service without running any worker(s). sudo service elasticsearch restart OR sudo service elasticsearch restart --force OR a system reboot

Environment

  1. OS: Ubuntu 20.04
  2. OpenCTI version: OpenCTI 5.3.7
  3. OpenCTI client: python
  4. Other environment details: Manual install

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Run the Mandiant connector script (only ran once) : python3 /opt/opencti/connectors/external-import/mandiant/src/mandiant.py
  2. Run the worker python script: python3 /opt/opencti/worker/worker.py &
  3. After importing a certain volume of IOCs, it crashes the Elasticsearch DB
  4. Errors:

Additional information

Elasticsearch Error:

{"category":"APP","error":{"context":{},"message":"connect ECONNREFUSED 127.0.0.1:9200","name":"ConnectionError","stack":"ConnectionError: connect ECONNREFUSED 127.0.0.1:9200\n at ClientRequest.onError (/sdb1/opt/opencti/build/node_modules/@elastic/elasticsearch/lib/Connection.js:123:16)\n at ClientRequest.emit (node:events:527:28)\n at Socket.socketErrorListener (node:_http_client:454:9)\n at Socket.emit (node:events:527:28)\n at emitErrorNT (node:internal/streams/destroy:157:8)\n at emitErrorCloseNT (node:internal/streams/destroy:122:3)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"level":"error","message":"[SEARCH ENGINE] Paginate fail","query":{"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"match_phrase":{"entity_type.keyword":"Sync"}},{"match_phrase":{"parent_types.keyword":"Sync"}}]}}],"must_not":[]}},"size":200,"sort":[{"standard_id.keyword":"asc"}]},"ignore_throttled":false,"index":["opencti_internal_objects*","opencti_stix_meta_objects*","opencti_stix_domain_objects*","opencti_stix_cyber_observables*","opencti_inferred_entities*"],"track_total_hits":true},"timestamp":"2022-09-06T10:29:32.637Z","version":"5.3.7"}

Worker Error: opencti_stix_core_relationships*%2Copencti_stix_sighting_relationships*%2Copencti_stix_cyber_observables*%2Copencti_stix_cyber_observable_relationships *%2Copencti_inferred_entities*%2Copencti_inferred_relationships*/_search","querystring":"size=5000&ignore_throttled=false","timeout":30000}}}," statusCode":null},"name":"ConnectionError"},"http_status":500,"query":{"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should": [{"term":{"internal_id.keyword":"rulemanager--32bc5b47-ecc0-5412-9bea-a789ab2e92a7"}},

faustus25 commented 1 year ago

Mandiant informed of need to separate IOC data and threat intel report data into two connectors to import separately.