search

OpenCTI-Platform / connectors

OpenCTI Connectors
https://www.opencti.io
Apache License 2.0
380 stars 412 forks source link

Elastic Stream Connector isn't Working #875

Open bakhtawarw opened 2 years ago

bakhtawarw commented 2 years ago

Description

Environment

  1. OS (where OpenCTI server runs): { Ubuntu 20.04 }
  2. OpenCTI version: { OpenCTI 5.3.15 }
  3. OpenCTI environment: Docker
  4. Elastic Security version: 7.15.0
  5. OpenCTI and Elastic Security are running on two separate ubuntu vms.

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Added elastic connector in OpenCTI's docker-compose file
  2. The connector is fed config.yml in the docker-compose file
  3. I am using Elastic CertUtil Self signed certs, and have linked them into the config.yml
  4. The connector is showing the following error:

elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, 'security_exception', 'action [indices:data/read/get] is unauthorized for API key id [SDVCmIMB-mxtRet2VslY] of user [elastic], this action is granted by the index privileges [read,all]') [2022-10-13T06:57:01.764][ERROR] Time since last heartbeat exceeded 45s, stopping the connector

Expected Output

Elastic Connector working correctly.

Actual Output

The connector is showing the following error.

image

Additional information

docker-compose file

image

Config.yml

opencti: url: 'http://10.0.2.7:8080' token: '53c59164-63c7-41ce-b097-56005536625a' ssl_verify: False connector: id: '47d39e8a-c6e1-4427-93d6-d74b4585fb60' type: 'STREAM' name: 'OpenCTI Elastic Connector' scope: 'elastic' confidence_level: 80 log_level: 'info' entity_name: 'Elastic Detection Cluster' entity_description: 'Elasticsearch detection engine cluster' live_stream_id: 'bc785509-52e5-4159-af3e-f148c6f181be' live_stream_listen_delete: true start_timestamp: '1655890657402' mode: ecs

output.elasticsearch: hosts: ["https://10.0.2.6:9200"] api_key: "SDVCmIMB-mxtRet2VslY:WDY5QGZ_RXKwMHq81CcVxQ" ssl_verify: false ssl_verify: false ssl.certificate_authorities: ["/app/certs/elasticsearch-ca.pem"] ssl.certificate: "/app/certs/ca.crt" ssl.key: "/app/certs/ca.key" setup: ilm: enabled: True policy_name: opencti rollover_alias: opencti pattern: "{now/d}-000001" overwrite: False template: enabled: True name: opencti pattern: opencti-* overwrite: False

elastic: signals: query_interval: '5m' lookback_interval: '5m' signal_index: '.siem-signals-*' query: > { "query": { "bool": { "must": { "match": { "signal.rule.type": "threat_match" } } } } }

I configured the api-key as follows:

POST /_security/api_key?pretty { "name": "opencti", "expiration": "365d", "role_descriptors": { "opencti_privileges": { "cluster": [ "monitor", "cluster:admin/ilm/get", "cluster:admin/ilm/put", "manage_index_templates" ], "indices": [ { "names": [ "opencti" ], "privileges": [ "all", "manage_follow_index" ], "field_security": { "grant": [ "" ] }, "allow_restricted_indices": false } ], "run_as": [] }, "protections_privileges": { "cluster": [], "indices": [ { "names": [ ".siem-signals-" ], "privileges": [ "read" ], "field_security": { "grant": [ "" ], "except": [] }, "allow_restricted_indices": false } ], "run_as": [] } }, "metadata": { "application": "opencti", "environment": { "tags": [ "dev", "staging" ] } } }

Screenshots (optional)

Elastic Stream

image

aaarghhh commented 2 years ago

Hi, I investigated about this issue. We got the same error when we tried to GET the document using wildcards *, in the update document phase of the connector. I was able to make a fix ( simply removing the wildcards parts and use alias instead ) but I haven't time to make proper tests. If someone will not fix it before, I'll try to PR it in the next week.

bakhtawarw commented 2 years ago

@dcode Can you please look into this issue?

bakhtawarw commented 2 years ago

@aaarghhh Can you please elaborate how did you fix it?

aaarghhh commented 2 years ago

This is my previous approach to maintain compatibility with the index. https://github.com/OpenCTI-Platform/connectors/commit/9465d2f23603add95792c4ec4834fd44bb106599 The main issue is that elastic get API doesn't supports wildcards, so I replaced with search but from my perspective it is an overhead.

I made some internal check and I figure out that the index could be referenced directly using its alias avoiding the wildcard. I have to test it, and I have to test my previous fix also.

bakhtawarw commented 2 years ago

@aaarghhh Hi, I tried your approach but it shows syntax errors. snip