No issue specific to this reported to this Connector.
Description
Manual install of OpenCTI (environment info)
Having ran the Mandiant connector to download content from the last 90 days the worker task never completes due to certain IOCs rendering in malformed STIX which crashes the Elasticsearch DB and requires a restart on the Elasticsearch service (sometimes forcing a service restart) and re-running OpenCTI.
The problem is start I cannot run any other connectors until the import finishes so I need to know how to stop the import.
Other connectors have ran fine without causing this issue.
The only workaround is too keep restarting the Elasticsearch service without running any worker(s).
sudo service elasticsearch restart OR sudo service elasticsearch restart --force OR a system reboot
Environment
OS: Ubuntu 20.04
OpenCTI version: OpenCTI 5.3.7
OpenCTI client: python
Other environment details: Manual install
Reproducible Steps
Steps to create the smallest reproducible scenario:
Run the Mandiant connector script (only ran once) : python3 /opt/opencti/connectors/external-import/mandiant/src/mandiant.py
Run the worker python script: python3 /opt/opencti/worker/worker.py &
After importing a certain volume of IOCs, it crashes the Elasticsearch DB
Errors:
Additional information
Elasticsearch Error:
{"category":"APP","error":{"context":{},"message":"connect ECONNREFUSED 127.0.0.1:9200","name":"ConnectionError","stack":"ConnectionError: connect ECONNREFUSED 127.0.0.1:9200\n at ClientRequest.onError (/sdb1/opt/opencti/build/node_modules/@elastic/elasticsearch/lib/Connection.js:123:16)\n at ClientRequest.emit (node:events:527:28)\n at Socket.socketErrorListener (node:_http_client:454:9)\n at Socket.emit (node:events:527:28)\n at emitErrorNT (node:internal/streams/destroy:157:8)\n at emitErrorCloseNT (node:internal/streams/destroy:122:3)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"level":"error","message":"[SEARCH ENGINE] Paginate fail","query":{"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"match_phrase":{"entity_type.keyword":"Sync"}},{"match_phrase":{"parent_types.keyword":"Sync"}}]}}],"must_not":[]}},"size":200,"sort":[{"standard_id.keyword":"asc"}]},"ignore_throttled":false,"index":["opencti_internal_objects*","opencti_stix_meta_objects*","opencti_stix_domain_objects*","opencti_stix_cyber_observables*","opencti_inferred_entities*"],"track_total_hits":true},"timestamp":"2022-09-06T10:29:32.637Z","version":"5.3.7"}
Prerequisites
No issue specific to this reported to this Connector.
Description
Manual install of OpenCTI (environment info)
Having ran the Mandiant connector to download content from the last 90 days the worker task never completes due to certain IOCs rendering in malformed STIX which crashes the Elasticsearch DB and requires a restart on the Elasticsearch service (sometimes forcing a service restart) and re-running OpenCTI.
The problem is start I cannot run any other connectors until the import finishes so I need to know how to stop the import. Other connectors have ran fine without causing this issue.
The only workaround is too keep restarting the Elasticsearch service without running any worker(s).
sudo service elasticsearch restart
ORsudo service elasticsearch restart --force
OR a system rebootEnvironment
Reproducible Steps
Steps to create the smallest reproducible scenario:
Additional information
Elasticsearch Error:
{"category":"APP","error":{"context":{},"message":"connect ECONNREFUSED 127.0.0.1:9200","name":"ConnectionError","stack":"ConnectionError: connect ECONNREFUSED 127.0.0.1:9200\n at ClientRequest.onError (/sdb1/opt/opencti/build/node_modules/@elastic/elasticsearch/lib/Connection.js:123:16)\n at ClientRequest.emit (node:events:527:28)\n at Socket.socketErrorListener (node:_http_client:454:9)\n at Socket.emit (node:events:527:28)\n at emitErrorNT (node:internal/streams/destroy:157:8)\n at emitErrorCloseNT (node:internal/streams/destroy:122:3)\n at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"level":"error","message":"[SEARCH ENGINE] Paginate fail","query":{"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should":[{"match_phrase":{"entity_type.keyword":"Sync"}},{"match_phrase":{"parent_types.keyword":"Sync"}}]}}],"must_not":[]}},"size":200,"sort":[{"standard_id.keyword":"asc"}]},"ignore_throttled":false,"index":["opencti_internal_objects*","opencti_stix_meta_objects*","opencti_stix_domain_objects*","opencti_stix_cyber_observables*","opencti_inferred_entities*"],"track_total_hits":true},"timestamp":"2022-09-06T10:29:32.637Z","version":"5.3.7"}
Worker Error:
opencti_stix_core_relationships*%2Copencti_stix_sighting_relationships*%2Copencti_stix_cyber_observables*%2Copencti_stix_cyber_observable_relationships *%2Copencti_inferred_entities*%2Copencti_inferred_relationships*/_search","querystring":"size=5000&ignore_throttled=false","timeout":30000}}}," statusCode":null},"name":"ConnectionError"},"http_status":500,"query":{"body":{"query":{"bool":{"must":[{"bool":{"minimum_should_match":1,"should": [{"term":{"internal_id.keyword":"rulemanager--32bc5b47-ecc0-5412-9bea-a789ab2e92a7"}},