OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.01k stars 892 forks source link

Search is broken when using OpenSearch and searching for string containing "-" #5678

Closed Jipegien closed 7 months ago

Jipegien commented 7 months ago

Description

When searching for a partial CVE number like cve-2024-3, platform is crashing.

image

Source: https://filigran-community.slack.com/archives/C06CF1N302W/p1706115440835929

Environment

  1. OS (where OpenCTI server runs):
  2. OpenCTI version: 5.12.21
  3. OpenCTI client:
  4. Other environment details: OpenSearch 2.11

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. On a platform running with OpenSearch 2.11
  2. search "cve-2024-3" (without doubling quote)

Expected Output

result or not

Actual Output

error

SamuelHassine commented 7 months ago

Tested and reproduced locally with OpenSearch last version, stack trace:

Reason: Query contains too many nested clauses; maxClauseCount is set to 1024

{"category":"APP","errors":[{"attributes":{"genre":"TECHNICAL","http_status":500,"query":{"_source":true,"body":{"query":{"bool":{"must":[{"bool":{"should":[{"bool":{"must_not":{"exists":{"field":"authorized_members"}}}},{"terms":{"authorized_members.id.keyword":["ALL","88ec0c6a-13ce-5e39-b486-354fe4a7084f",
"e7368846-fb3a-45a6-b712-5833e1114c4a","a2f8b969-278a-4a53-a7fd-cbbd1dcc8eb9"]}}]}},{"bool":{"minimum_should_match":2,"should":[{"bool":{"minimum_should_match":1,"should":[{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["entity_type.keyword","parent_types.keyword"],"query":"Stix-Core-Ob
ject"}}]}}]}},{"bool":{"minimum_should_match":1,"should":[{"multi_match":{"fields":["entity_type.keyword","parent_types.keyword"],"query":"Stix-Core-Object"}}]}}]}},{"bool":{"minimum_should_match":1,"should":[{"query_string":{"analyze_wildcard":true,"fields":["name^5","description^2","attribute_abstract^5","
explanation^5","id","internal_id","standard_id","parent_types","base_type","entity_type","creator_id","x_opencti_stix_ids","x_opencti_files.id","x_opencti_files.name","x_opencti_files.description","x_opencti_files.mime_type","lang","x_opencti_graph_data","x_opencti_workflow_id","contact_information","roles",
"identity_class","x_opencti_location_type","aliases","i_aliases_ids","x_mitre_platforms","x_mitre_permissions_required","x_mitre_detection","x_mitre_id","objective","content","authors","note_types","content_mapping","opinion","report_types","x_opencti_reliability","x_opencti_aliases","x_opencti_threat_huntin
g","x_opencti_log_sources","x_opencti_firstname","x_opencti_lastname","infrastructure_types","goals","resource_level","primary_motivation","secondary_motivations","postal_code","street_address","malware_types","architecture_execution_envs","implementation_languages","capabilities","threat_actor_types","sophi
stication","personal_motivations","tool_types","tool_version","x_opencti_base_severity","x_opencti_attack_vector","x_opencti_integrity_impact","x_opencti_availability_impact","x_opencti_confidentiality_impact","incident_type","severity","source","platform_title","platform_organization","platform_favicon","pl
atform_email","platform_theme","platform_theme_dark_background","platform_theme_dark_paper","platform_theme_dark_nav","platform_theme_dark_primary","platform_theme_dark_secondary","platform_theme_dark_accent","platform_theme_dark_logo","platform_theme_dark_logo_collapsed","platform_theme_dark_logo_login","pl
atform_theme_light_background","platform_theme_light_paper","platform_theme_light_nav","platform_theme_light_primary","platform_theme_light_secondary","platform_theme_light_accent","platform_theme_light_logo","platform_theme_light_logo_collapsed","platform_theme_light_logo_login","platform_language","platfor
m_login_message","platform_consent_message","platform_consent_confirm_text","platform_banner_text","platform_banner_level","activity_listeners_ids","platform_messages","analytics_google_analytics_v4","lastRun","platformVersion","title","default_marking.entity_type","default_marking.values","default_dashboard
","default_hidden_types","user_email","password","firstname","lastname","theme","language","bookmarks.id","bookmarks.type","api_token","otp_secret","otp_qr","default_time_field","account_status","administrated_organizations","unit_system","lastEventId","errors.id","errors.message","errors.error","errors.sour
ce","connector_type","connector_scope","connector_state","connector_user_id","filters","authorized_members.id","authorized_members.name","authorized_members.entity_type","authorized_members.access_right","separator","feed_types","feed_date_attribute","feed_attributes.attribute","mappings.type","mappings.attr
ibute","color","template_id","type","event_source_id","event_type","user_id","connector_id","status","messages.message","scope","rule","initiator_id","task_filters","task_search","task_position","task_ids","task_excluded_ids","uri","token","stream_id","event_status","event_access","event_scope","applicant_id
","group_ids","organization_ids","context_data.id","context_data.provider","context_data.username","context_data.message","context_data.commit","context_data.element_id","context_data.entity_type","context_data.path","context_data.format","context_data.operation","context_data.entity_name","context_data.expo
rt_scope","context_data.export_type","context_data.file_id","context_data.file_name","context_data.file_mime","context_data.max_marking","context_data.connectors","context_data.selected_ids","context_data.connector_name","context_data.marking_definition_ids","context_data.object_marking_refs_ids","context_da
ta.from_id","context_data.to_id","context_data.created_by_id","context_data.created_by_ref_id","context_data.creator_ids","context_data.granted_refs_ids","context_data.labels_ids","context_data.workspace_type","context_data.types","context_data.search","context_data.filters","relationship_type","fromType","t
oType","connections.internal_id","connections.name","connections.role","connections.types","grant","x_opencti_description","rir","path","path_enc","value","display_name","content_type","message_id","subject","received_lines","body","content_disposition","hashes.MD5","hashes.SHA-1","hashes.SHA-256","hashes.SH
A-512","hashes.SHA3-256","hashes.SHA3-512","hashes.SSDEEP","hashes.SDHASH","hashes.TLSH","hashes.LZJD","mime_type","payload_bin","url","encryption_algorithm","decryption_key","x_opencti_additional_names","extensions","name_enc","magic_number_hex","obsContent","version","serial_number","signature_algorithm","
issuer","subject_public_key_algorithm","subject_public_key_modulus","basic_constraints","name_constraints","policy_constraints","key_usage","extended_key_usage","subject_key_identifier","authority_key_identifier","subject_alternative_name","issuer_alternative_name","subject_directory_attributes","crl_distrib
ution_points","inhibit_any_policy","certificate_policies","policy_mappings","protocols","cwd","command_line","environment_variables","priority","owner_sid","window_title","integrity_level","service_name","descriptions","group_name","start_type","service_type","service_status","cpe","swid","languages","vendor
","credential","account_login","account_type","attribute_key","data","data_type","iban","bic","account_number","card_number","holder_name","media_category","definition_type","definition","x_opencti_color","source_name","hash","external_id","kill_chain_name","phase_name","channel_types","event_types","context
","narrative_types","trigger_scope","outcomes","notifiers","recipients","trigger_ids","period","trigger_time","trigger_type","authorized_authorities","notification_type","notification_content.title","events.message","events.instance_id","events.operation","collection_layers","category","caseTemplate","respon
se_types","information_types","takedown_types","target_type","attributes_configuration","availableSettings","openCTI_version","manifest","tags","graph_data","investigated_entities_ids","product","configuration_version","modules","analysis_engine_version","analysis_definition_version","result_name","result","
manager_id","notifier_connector_id","notifier_configuration","gender","job_title","marital_status","eye_color","hair_color","playbook_start","playbook_definition","created_by_ref","object_marking_refs","collection","authentication_type","authentication_value","current_state_cursor","pattern_type","pattern_ve
rsion","pattern","indicator_types","x_opencti_main_observable_type","x_opencti_organization_type","grantable_groups","representations","skipLineChar","information","uploadStatus","metaData.description","metaData.list_filters","metaData.filename","metaData.mimetype","metaData.labels_text","metaData.labels","m
etaData.encoding","metaData.creator_id","metaData.entity_id","metaData.external_reference_id","attachment.author","attachment.comments","attachment.content","attachment.content_type","attachment.creator_tool","attachment.description","attachment.format","attachment.keywords","attachment.language","attachment
.modifier","attachment.title","file_id","entity_id"],"query":"cve\\-2024\\-03*"}},{"multi_match":{"fields":["name^5","description^2","attribute_abstract^5","explanation^5","id","internal_id","standard_id","parent_types","base_type","entity_type","creator_id","x_opencti_stix_ids","x_opencti_files.id","x_openc
ti_files.name","x_opencti_files.description","x_opencti_files.mime_type","lang","x_opencti_graph_data","x_opencti_workflow_id","contact_information","roles","identity_class","x_opencti_location_type","aliases","i_aliases_ids","x_mitre_platforms","x_mitre_permissions_required","x_mitre_detection","x_mitre_id"
,"objective","content","authors","note_types","content_mapping","opinion","report_types","x_opencti_reliability","x_opencti_aliases","x_opencti_threat_hunting","x_opencti_log_sources","x_opencti_firstname","x_opencti_lastname","infrastructure_types","goals","resource_level","primary_motivation","secondary_mo
tivations","postal_code","street_address","malware_types","architecture_execution_envs","implementation_languages","capabilities","threat_actor_types","sophistication","personal_motivations","tool_types","tool_version","x_opencti_base_severity","x_opencti_attack_vector","x_opencti_integrity_impact","x_openct
i_availability_impact","x_opencti_confidentiality_impact","incident_type","severity","source","platform_title","platform_organization","platform_favicon","platform_email","platform_theme","platform_theme_dark_background","platform_theme_dark_paper","platform_theme_dark_nav","platform_theme_dark_primary","pla
tform_theme_dark_secondary","platform_theme_dark_accent","platform_theme_dark_logo","platform_theme_dark_logo_collapsed","platform_theme_dark_logo_login","platform_theme_light_background","platform_theme_light_paper","platform_theme_light_nav","platform_theme_light_primary","platform_theme_light_secondary","
platform_theme_light_accent","platform_theme_light_logo","platform_theme_light_logo_collapsed","platform_theme_light_logo_login","platform_language","platform_login_message","platform_consent_message","platform_consent_confirm_text","platform_banner_text","platform_banner_level","activity_listeners_ids","pla
tform_messages","analytics_google_analytics_v4","lastRun","platformVersion","title","default_marking.entity_type","default_marking.values","default_dashboard","default_hidden_types","user_email","password","firstname","lastname","theme","language","bookmarks.id","bookmarks.type","api_token","otp_secret","otp
_qr","default_time_field","account_status","administrated_organizations","unit_system","lastEventId","errors.id","errors.message","errors.error","errors.source","connector_type","connector_scope","connector_state","connector_user_id","filters","authorized_members.id","authorized_members.name","authorized_mem
bers.entity_type","authorized_members.access_right","separator","feed_types","feed_date_attribute","feed_attributes.attribute","mappings.type","mappings.attribute","color","template_id","type","event_source_id","event_type","user_id","connector_id","status","messages.message","scope","rule","initiator_id","t
ask_filters","task_search","task_position","task_ids","task_excluded_ids","uri","token","stream_id","event_status","event_access","event_scope","applicant_id","group_ids","organization_ids","context_data.id","context_data.provider","context_data.username","context_data.message","context_data.commit","context
_data.element_id","context_data.entity_type","context_data.path","context_data.format","context_data.operation","context_data.entity_name","context_data.export_scope","context_data.export_type","context_data.file_id","context_data.file_name","context_data.file_mime","context_data.max_marking","context_data.c
onnectors","context_data.selected_ids","context_data.connector_name","context_data.marking_definition_ids","context_data.object_marking_refs_ids","context_data.from_id","context_data.to_id","context_data.created_by_id","context_data.created_by_ref_id","context_data.creator_ids","context_data.granted_refs_ids
","context_data.labels_ids","context_data.workspace_type","context_data.types","context_data.search","context_data.filters","relationship_type","fromType","toType","connections.internal_id","connections.name","connections.role","connections.types","grant","x_opencti_description","rir","path","path_enc","valu
e","display_name","content_type","message_id","subject","received_lines","body","content_disposition","hashes.MD5","hashes.SHA-1","hashes.SHA-256","hashes.SHA-512","hashes.SHA3-256","hashes.SHA3-512","hashes.SSDEEP","hashes.SDHASH","hashes.TLSH","hashes.LZJD","mime_type","payload_bin","url","encryption_algor
ithm","decryption_key","x_opencti_additional_names","extensions","name_enc","magic_number_hex","obsContent","version","serial_number","signature_algorithm","issuer","subject_public_key_algorithm","subject_public_key_modulus","basic_constraints","name_constraints","policy_constraints","key_usage","extended_ke
y_usage","subject_key_identifier","authority_key_identifier","subject_alternative_name","issuer_alternative_name","subject_directory_attributes","crl_distribution_points","inhibit_any_policy","certificate_policies","policy_mappings","protocols","cwd","command_line","environment_variables","priority","owner_s
id","window_title","integrity_level","service_name","descriptions","group_name","start_type","service_type","service_status","cpe","swid","languages","vendor","credential","account_login","account_type","attribute_key","data","data_type","iban","bic","account_number","card_number","holder_name","media_catego
ry","definition_type","definition","x_opencti_color","source_name","hash","external_id","kill_chain_name","phase_name","channel_types","event_types","context","narrative_types","trigger_scope","outcomes","notifiers","recipients","trigger_ids","period","trigger_time","trigger_type","authorized_authorities","n
otification_type","notification_content.title","events.message","events.instance_id","events.operation","collection_layers","category","caseTemplate","response_types","information_types","takedown_types","target_type","attributes_configuration","availableSettings","openCTI_version","manifest","tags","graph_d
ata","investigated_entities_ids","product","configuration_version","modules","analysis_engine_version","analysis_definition_version","result_name","result","manager_id","notifier_connector_id","notifier_configuration","gender","job_title","marital_status","eye_color","hair_color","playbook_start","playbook_d
efinition","created_by_ref","object_marking_refs","collection","authentication_type","authentication_value","current_state_cursor","pattern_type","pattern_version","pattern","indicator_types","x_opencti_main_observable_type","x_opencti_organization_type","grantable_groups","representations","skipLineChar","i
nformation","uploadStatus","metaData.description","metaData.list_filters","metaData.filename","metaData.mimetype","metaData.labels_text","metaData.labels","metaData.encoding","metaData.creator_id","metaData.entity_id","metaData.external_reference_id","attachment.author","attachment.comments","attachment.cont
ent","attachment.content_type","attachment.creator_tool","attachment.description","attachment.format","attachment.keywords","attachment.language","attachment.modifier","attachment.title","file_id","entity_id"],"lenient":true,"query":"cve\\-2024\\-03*","type":"phrase"}},{"nested":{"path":"connections","query"
:{"bool":{"must":[{"query_string":{"analyze_wildcard":true,"fields":["connections.name^5","connections.*"],"query":"cve\\-2024\\-03*"}}]}}}}]}}],"must_not":[]}},"size":25,"sort":[{"_score":"desc"},{"standard_id.keyword":"asc"}]},"ignore_throttled":false,"index":["opencti_internal_objects*","opencti_stix_meta
_objects*","opencti_stix_domain_objects*","opencti_stix_cyber_observables*","opencti_inferred_entities*"],"track_total_hits":true}},"message":"Fail to execute engine pagination","name":"DATABASE_ERROR","stack":"DATABASE_ERROR: Fail to execute engine pagination\n    at error (C:\\Users\\SamuelHassine\\Dropbox
\\Developpement\\Associations\\Luatix\\Produits\\OpenCTI\\opencti-app\\opencti-platform\\opencti-graphql\\src\\config\\errors.js:8:10)\n    at DatabaseError (C:\\Users\\SamuelHassine\\Dropbox\\Developpement\\Associations\\Luatix\\Produits\\OpenCTI\\opencti-app\\opencti-platform\\opencti-graphql\\src\\config\
\errors.js:58:48)\n    at C:\\Users\\SamuelHassine\\Dropbox\\Developpement\\Associations\\Luatix\\Produits\\OpenCTI\\opencti-app\\opencti-platform\\opencti-graphql\\src\\database\\engine.js:2483:15\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)"},{"message":"search_phase_execution
_exception: [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCoun
t is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024","name":"ResponseError","stack":"ResponseError: search_phase_execution_exception: 
[too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 
1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024; [too_many_nested_clauses] Reason: Query contains too many nested clauses; maxClauseCount is set to 1024\n    at onBody (C:\\Users\\SamuelHassine\\Dropbox\\Developpement\\Associations\\Luatix\\Produi
ts\\OpenCTI\\opencti-app\\opencti-platform\\opencti-graphql\\node_modules\\@opensearch-project\\opensearch\\lib\\Transport.js:425:23)\n    at IncomingMessage.onEnd (C:\\Users\\SamuelHassine\\Dropbox\\Developpement\\Associations\\Luatix\\Produits\\OpenCTI\\opencti-app\\opencti-platform\\opencti-graphql\\node_
modules\\@opensearch-project\\opensearch\\lib\\Transport.js:340:11)\n    at IncomingMessage.emit (node:events:524:35)\n    at endReadableNT (node:internal/streams/readable:1378:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"}],"inner_relation_creation":0,"level":"error","mess
age":"Fail to execute engine pagination","operation":"SearchStixCoreObjectsLinesPaginationQuery","operation_query":"query SearchStixCoreObjectsLinesPaginationQuery($types:[String]$search:String$count:Int!$cursor:ID$orderBy:StixCoreObjectsOrdering$orderMode:OrderingMode$filters:FilterGroup){...SearchStixCoreO
bjectsLines_data_4GmerJ}fragment SearchStixCoreObjectLine_node on StixCoreObject{__isStixCoreObject:__typename id parent_types entity_type created_at ...on AttackPattern{name description aliases}...on Campaign{name description aliases}...on Note{attribute_abstract content}...on ObservedData{name first_observ
ed last_observed}...on Opinion{opinion explanation}...on Report{name description}...on Grouping{name description}...on CourseOfAction{name description x_opencti_aliases}...on Individual{name description x_opencti_aliases}...on Organization{name description x_opencti_aliases}...on Sector{name description x_op
encti_aliases}...on System{name description x_opencti_aliases}...on Indicator{name description}...on Infrastructure{name description}...on IntrusionSet{name aliases description}...on Position{name description x_opencti_aliases}...on City{name description x_opencti_aliases}...on AdministrativeArea{name descri
ption x_opencti_aliases}...on Country{name description x_opencti_aliases}...on Region{name description x_opencti_aliases}...on Malware{name aliases description}...on ThreatActor{__isThreatActor:__typename name aliases description}...on Tool{name aliases description}...on Vulnerability{name description}...on 
Incident{name aliases description}...on Event{name description aliases}...on Channel{name description aliases}...on Narrative{name description aliases}...on Language{name aliases}...on DataComponent{name}...on DataSource{name}...on Case{__isCase:__typename name}...on StixCyberObservable{__isStixCyberObservab
le:__typename observable_value}...on StixFile{x_opencti_additional_names}...on IPv4Addr{countries{edges{node{name x_opencti_aliases id}}}}...on IPv6Addr{countries{edges{node{name x_opencti_aliases id}}}}createdBy{__typename __isIdentity:__typename name id}objectMarking{edges{node{id definition_type definitio
n x_opencti_order x_opencti_color}}}objectLabel{edges{node{id value color}}}creators{id name}containersNumber{total}}fragment SearchStixCoreObjectsLines_data_4GmerJ on Query{globalSearch(types:$types search:$search first:$count after:$cursor orderBy:$orderBy orderMode:$orderMode filters:$filters){edges{node{
__typename id entity_type created_at createdBy{__typename __isIdentity:__typename name id}creators{id name}objectMarking{edges{node{id definition_type definition x_opencti_order x_opencti_color}}}...SearchStixCoreObjectLine_node}cursor}pageInfo{endCursor hasNextPage globalCount}}}","size":208,"time":24,"time
stamp":"2024-01-25T08:16:40.432Z","type":"READ_ERROR","user":{"group_ids":["e7368846-fb3a-45a6-b712-5833e1114c4a"],"ip":"::1","organization_ids":[],"referer":"http://localhost:3000/dashboard/search/knowledge/cve-2024-03?filters=%7B%22mode%22%3A%22and%22%2C%22filters%22%3A%5B%7B%22id%22%3A%22251a939b-8269-40c
d-94d2-827a9682924f%22%2C%22key%22%3A%22entity_type%22%2C%22values%22%3A%5B%5D%2C%22operator%22%3A%22eq%22%2C%22mode%22%3A%22or%22%7D%5D%2C%22filterGroups%22%3A%5B%5D%7D&sortBy=_score&orderAsc=false","socket":"query","user_id":"88ec0c6a-13ce-5e39-b486-354fe4a7084f","user_metadata":{}},"variables":{"count":25,"filters":{"filterGroups":[],"filters":[{"key":"entity_type","mode":"or","operator":"eq","values":["Stix-Core-Object"]}],"mode":"and"},"orderBy":"_score","orderMode":"desc","search":"cve-2024-03"},"version":"5.12.21"}
nino-filigran commented 7 months ago

Same as https://github.com/OpenCTI-Platform/opencti/issues/5673

Jipegien commented 7 months ago

string with "-" are considered as multiple words to search in Elastic or OpenSearch. OpenSearch has a limit of 1024 Clauses per search and we reach this limit with the global search if more than 2 words are entered. We need to find a way to limit the number of Clauses needed in this type of search. In any case, limit may still be reached (either with OpenSearch or Elastic, but the limit of Elastic is way bigger) with long sentences. So we need to catch the OpenSearch/Elastic error and display an adequate error message in the UI.

richard-julien commented 7 months ago

2 things done: