"ERR Find direct ids fail" on malware Knowledge overview tab.

[FIying-Scotsman]

Description

After importing 3 PDFs under a malware profile, viewing the Overview tab under Knowledge shows a "An unknown error occurred. Please contact your administrator or the OpenCTI maintainers." error after a couple of seconds.

Environment

1. OS (where OpenCTI server runs): Ubuntu
2. OpenCTI version: 6.0.7
3. OpenCTI client: 6.0.7
4. Other environment details: Mitre & VirusTotal connectors present; 4 cores/8Gb of RAM.

Reproducible Steps

Steps to create the smallest reproducible scenario:

1. Create a malware profile under the Malware section.
2. With Mitre data present, import the following 3 PDFs under "Data":-

• https://info.spamhaus.com/hubfs/Botnet%20Reports/2023%20Q3%20Botnet%20Threat%20Update.pdf
• https://any.run/cybersecurity-blog/crackedcantil-breakdown/
• https://github.com/echocti/ECHO-Reports/blob/main/Malware%20Analysis%20Report/StealC/StealC_Technical_Analysis_Report.pdf

4. Under workbenches, remove Amazon and Google.com Observables via multi-select & delete using delete button (bottom right).

Expected Output

Knowledge page shouldn't cause the platform to crash and spam the container log file with the below log.

Actual Output

Knowledge overview page crashes the platform. All other tabs load fine - just the timeline section causes the crash.

Additional information

Sanitised log which is repeated over and over:-

1970/01/01 01: 20AM ERR Find direct ids fail | category=APP errors=[
    {
        "attributes": {
            "genre": "TECHNICAL",
            "http_status": 500,
            "query": {
                "_source": true,
                "body": {
                    "query": {
                        "bool": {
                            "must": [
                                {
                                    "bool": {
                                        "minimum_should_match": 1,
                                        "should": [
                                            {
                                                "terms": {
                                                    "internal_id.keyword": [
                                                        "95a9f685-02c4-4cc9-a3ab-71cced90d051"
                                                    ]
                                                }
                                            },
                                            {
                                                "terms": {
                                                    "standard_id.keyword": [
                                                        "95a9f685-02c4-4cc9-a3ab-71cced90d051"
                                                    ]
                                                }
                                            },
                                            {
                                                "terms": {
                                                    "x_opencti_stix_ids.keyword": [
                                                        "95a9f685-02c4-4cc9-a3ab-71cced90d051"
                                                    ]
                                                }
                                            }
                                        ]
                                    }
                                }
                            ],
                            "must_not": []
                        }
                    },
                    "sort": [
                        {
                            "created_at": "asc"
                        }
                    ]
                },
                "index": [
                    "opencti_internal_objects*",
                    "opencti_internal_relationships*",
                    "opencti_stix_meta_objects*",
                    "opencti_stix_meta_relationships*",
                    "opencti_stix_cyber_observable_relationships*",
                    "opencti_stix_domain_objects*",
                    "opencti_stix_core_relationships*",
                    "opencti_stix_sighting_relationships*",
                    "opencti_stix_cyber_observables*",
                    "opencti_inferred_entities*",
                    "opencti_inferred_relationships*"
                ],
                "size": 1
            }
        },
        "message": "Find direct ids fail",
        "name": "DATABASE_ERROR",
        "stack": "DATABASE_ERROR: Find direct ids fail\n    at error (/opt/opencti/build/src/config/errors.js:8:10)\n    at DatabaseError (/opt/opencti/build/src/config/errors.js:58:48)\n    at /opt/opencti/build/src/database/engine.js:1311:15\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1310:20)\n    at internalFindByIds (/opt/opencti/build/src/database/middleware-loader.ts:537:10)\n    at batchInternalRels (/opt/opencti/build/src/domain/stixCoreObject.js:83:28)"
    },
    {
        "message": "search_phase_execution_exception\n\tCaused by:\n\t\tes_rejected_execution_exception: rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@4ea1e859, creationTimeNanos=262858583476044, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = 28c78391e30a/search, queue capacity = 1000, task execution EWMA = 614.6micros, total task execution time = 2.3h, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@722d3ddb[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 6125181]]",
        "name": "ResponseError",
        "stack": "ResponseError: search_phase_execution_exception\n\tCaused by:\n\t\tes_rejected_execution_exception: rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@4ea1e859, creationTimeNanos=262858583476044, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = 28c78391e30a/search, queue capacity = 1000, task execution EWMA = 614.6micros, total task execution time = 2.3h, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@722d3ddb[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 6125181]]\n    at dsn.request (/opt/opencti/build/node_modules/@elastic/transport/src/Transport.ts:553:17)\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at Ort.SearchApi [as search] (/opt/opencti/build/node_modules/@elastic/elasticsearch/src/api/api/search.ts:89:10)\n    at elFindByIds (/opt/opencti/build/src/database/engine.js:1310:20)\n    at internalFindByIds (/opt/opencti/build/src/database/middleware-loader.ts:537:10)\n    at batchInternalRels (/opt/opencti/build/src/domain/stixCoreObject.js:83:28)"
    }
] inner_relation_creation=0 operation=StixDomainObjectThreatKnowledgeQueryStixRelationshipsQuery operation_query=query StixDomainObjectThreatKnowledgeQueryStixRelationshipsQuery($fromOrToId:String$elementWithTargetTypes: [String
]$relationship_type: [String
]$first:Int$orderBy:StixRelationshipsOrdering$orderMode:OrderingMode$filters:FilterGroup){...StixDomainObjectGlobalKillChain_data ...StixDomainObjectTimeline_data
}fragment StixDomainObjectGlobalKillChain_data on Query{stixRelationships(fromOrToId:$fromOrToId elementWithTargetTypes:$elementWithTargetTypes relationship_type:$relationship_type first:$first orderBy:$orderBy orderMode:$orderMode filters:$filters){edges{node{__typename id entity_type ...on StixCoreRelationship{description created start_time stop_time killChainPhases{id phase_name x_opencti_order
                    }objectMarking{id definition_type definition x_opencti_order x_opencti_color
                    }
                }to{__typename ...on BasicObject{__isBasicObject:__typename id entity_type
                    }...on AttackPattern{name x_mitre_id killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Campaign{name id
                    }...on CourseOfAction{name id
                    }...on Individual{name id
                    }...on Organization{name id
                    }...on Sector{name id
                    }...on System{name id
                    }...on Indicator{name id
                    }...on Infrastructure{name id
                    }...on IntrusionSet{name id
                    }...on Position{name id
                    }...on City{name id
                    }...on AdministrativeArea{name id
                    }...on Country{name id
                    }...on Region{name id
                    }...on Malware{name killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on ThreatActor{__isThreatActor:__typename name
                    }...on Tool{name killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Vulnerability{name id
                    }...on Incident{name id
                    }...on Artifact{id
                    }...on AutonomousSystem{id
                    }...on BankAccount{id
                    }...on CaseIncident{id
                    }...on CaseRfi{id
                    }...on CaseRft{id
                    }...on CaseTemplate{id
                    }...on Channel{id
                    }...on Creator{id
                    }...on CryptocurrencyWallet{id
                    }...on CryptographicKey{id
                    }...on CsvMapper{id
                    }...on DataComponent{id
                    }...on DataSource{id
                    }...on Directory{id
                    }...on DomainName{id
                    }...on EmailAddr{id
                    }...on EmailMessage{id
                    }...on EmailMimePartType{id
                    }...on EntitySetting{id
                    }...on Event{id
                    }...on ExternalReference{id
                    }...on Feedback{id
                    }...on Group{id
                    }...on Grouping{id
                    }...on Hostname{id
                    }...on IPv4Addr{id
                    }...on IPv6Addr{id
                    }...on KillChainPhase{id
                    }...on Label{id
                    }...on Language{id
                    }...on MacAddr{id
                    }...on MalwareAnalysis{id
                    }...on ManagerConfiguration{id
                    }...on MarkingDefinition{id
                    }...on MediaContent{id
                    }...on Mutex{id
                    }...on Narrative{id
                    }...on NetworkTraffic{id
                    }...on Note{id
                    }...on ObservedData{id
                    }...on Opinion{id
                    }...on PaymentCard{id
                    }...on PhoneNumber{id
                    }...on Process{id
                    }...on PublicDashboard{id
                    }...on Report{id
                    }...on Software{id
                    }...on Status{id
                    }...on StixCoreRelationship{id
                    }...on StixFile{id
                    }...on StixRefRelationship{id
                    }...on StixSightingRelationship{id
                    }...on Task{id
                    }...on Text{id
                    }...on ThreatActorGroup{id
                    }...on ThreatActorIndividual{id
                    }...on Url{id
                    }...on UserAccount{id
                    }...on UserAgent{id
                    }...on WindowsRegistryKey{id
                    }...on WindowsRegistryValueType{id
                    }...on Workspace{id
                    }...on X509Certificate{id
                    }
                }from{__typename ...on BasicObject{__isBasicObject:__typename id entity_type
                    }...on AttackPattern{name x_mitre_id killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Campaign{name id
                    }...on CourseOfAction{name id
                    }...on Individual{name id
                    }...on Organization{name id
                    }...on Sector{name id
                    }...on System{name id
                    }...on Indicator{name id
                    }...on Infrastructure{name id
                    }...on IntrusionSet{name id
                    }...on Position{name id
                    }...on City{name id
                    }...on AdministrativeArea{name id
                    }...on Country{name id
                    }...on Region{name id
                    }...on Malware{name killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on ThreatActor{__isThreatActor:__typename name
                    }...on Tool{name killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Vulnerability{name id
                    }...on Incident{name id
                    }...on Artifact{id
                    }...on AutonomousSystem{id
                    }...on BankAccount{id
                    }...on CaseIncident{id
                    }...on CaseRfi{id
                    }...on CaseRft{id
                    }...on CaseTemplate{id
                    }...on Channel{id
                    }...on Creator{id
                    }...on CryptocurrencyWallet{id
                    }...on CryptographicKey{id
                    }...on CsvMapper{id
                    }...on DataComponent{id
                    }...on DataSource{id
                    }...on Directory{id
                    }...on DomainName{id
                    }...on EmailAddr{id
                    }...on EmailMessage{id
                    }...on EmailMimePartType{id
                    }...on EntitySetting{id
                    }...on Event{id
                    }...on ExternalReference{id
                    }...on Feedback{id
                    }...on Group{id
                    }...on Grouping{id
                    }...on Hostname{id
                    }...on IPv4Addr{id
                    }...on IPv6Addr{id
                    }...on KillChainPhase{id
                    }...on Label{id
                    }...on Language{id
                    }...on MacAddr{id
                    }...on MalwareAnalysis{id
                    }...on ManagerConfiguration{id
                    }...on MarkingDefinition{id
                    }...on MediaContent{id
                    }...on Mutex{id
                    }...on Narrative{id
                    }...on NetworkTraffic{id
                    }...on Note{id
                    }...on ObservedData{id
                    }...on Opinion{id
                    }...on PaymentCard{id
                    }...on PhoneNumber{id
                    }...on Process{id
                    }...on PublicDashboard{id
                    }...on Report{id
                    }...on Software{id
                    }...on Status{id
                    }...on StixCoreRelationship{id
                    }...on StixFile{id
                    }...on StixRefRelationship{id
                    }...on StixSightingRelationship{id
                    }...on Task{id
                    }...on Text{id
                    }...on ThreatActorGroup{id
                    }...on ThreatActorIndividual{id
                    }...on Url{id
                    }...on UserAccount{id
                    }...on UserAgent{id
                    }...on WindowsRegistryKey{id
                    }...on WindowsRegistryValueType{id
                    }...on Workspace{id
                    }...on X509Certificate{id
                    }
                }
            }
        }
    }
}fragment StixDomainObjectTimeline_data on Query{stixRelationships(fromOrToId:$fromOrToId elementWithTargetTypes:$elementWithTargetTypes relationship_type:$relationship_type first:$first orderBy:$orderBy orderMode:$orderMode filters:$filters){edges{node{__typename id entity_type parent_types ...on StixRefRelationship{created_at
                }...on StixCoreRelationship{description created start_time stop_time killChainPhases{id phase_name x_opencti_order
                    }objectMarking{id definition_type definition x_opencti_order x_opencti_color
                    }
                }...on StixSightingRelationship{created first_seen last_seen objectMarking{id definition_type definition x_opencti_order x_opencti_color
                    }
                }from{__typename ...on BasicObject{__isBasicObject:__typename id entity_type
                    }...on AttackPattern{name description x_mitre_id killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Campaign{name description id
                    }...on CourseOfAction{name description id
                    }...on Note{attribute_abstract content id
                    }...on Opinion{opinion id
                    }...on Individual{name description id
                    }...on Organization{name description id
                    }...on Sector{name description id
                    }...on System{name description id
                    }...on Indicator{name description id
                    }...on Infrastructure{name description id
                    }...on IntrusionSet{name description id
                    }...on Position{name description id
                    }...on City{name description id
                    }...on AdministrativeArea{name description id
                    }...on Country{name description id
                    }...on Region{name description id
                    }...on Malware{name description killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on MalwareAnalysis{result_name id
                    }...on ThreatActor{__isThreatActor:__typename name description
                    }...on Tool{name description killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Vulnerability{name description id
                    }...on Incident{name description id
                    }...on StixCyberObservable{__isStixCyberObservable:__typename x_opencti_description observable_value
                    }...on Event{name description id
                    }...on Channel{name description id
                    }...on Narrative{name description id
                    }...on Language{name id
                    }...on DataComponent{name id
                    }...on DataSource{name id
                    }...on Case{__isCase:__typename name
                    }...on Report{name id
                    }...on Grouping{name id
                    }...on ObservedData{name id
                    }...on Creator{name id
                    }...on MarkingDefinition{definition id
                    }...on ExternalReference{source_name url description id
                    }...on Artifact{id
                    }...on AutonomousSystem{id
                    }...on BankAccount{id
                    }...on CaseIncident{id
                    }...on CaseRfi{id
                    }...on CaseRft{id
                    }...on CaseTemplate{id
                    }...on CryptocurrencyWallet{id
                    }...on CryptographicKey{id
                    }...on CsvMapper{id
                    }...on Directory{id
                    }...on DomainName{id
                    }...on EmailAddr{id
                    }...on EmailMessage{id
                    }...on EmailMimePartType{id
                    }...on EntitySetting{id
                    }...on Feedback{id
                    }...on Group{id
                    }...on Hostname{id
                    }...on IPv4Addr{id
                    }...on IPv6Addr{id
                    }...on KillChainPhase{id
                    }...on Label{id
                    }...on MacAddr{id
                    }...on ManagerConfiguration{id
                    }...on MediaContent{id
                    }...on Mutex{id
                    }...on NetworkTraffic{id
                    }...on PaymentCard{id
                    }...on PhoneNumber{id
                    }...on Process{id
                    }...on PublicDashboard{id
                    }...on Software{id
                    }...on Status{id
                    }...on StixCoreRelationship{id
                    }...on StixFile{id
                    }...on StixRefRelationship{id
                    }...on StixSightingRelationship{id
                    }...on Task{id
                    }...on Text{id
                    }...on ThreatActorGroup{id
                    }...on ThreatActorIndividual{id
                    }...on Url{id
                    }...on UserAccount{id
                    }...on UserAgent{id
                    }...on WindowsRegistryKey{id
                    }...on WindowsRegistryValueType{id
                    }...on Workspace{id
                    }...on X509Certificate{id
                    }
                }to{__typename ...on BasicObject{__isBasicObject:__typename id entity_type
                    }...on AttackPattern{name description x_mitre_id killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Campaign{name description id
                    }...on Note{attribute_abstract content id
                    }...on Opinion{opinion id
                    }...on CourseOfAction{name description id
                    }...on Individual{name description id
                    }...on Organization{name description id
                    }...on Sector{name description id
                    }...on System{name description id
                    }...on Indicator{name description id
                    }...on Infrastructure{name description id
                    }...on IntrusionSet{name description id
                    }...on Position{name description id
                    }...on City{name description id
                    }...on AdministrativeArea{name description id
                    }...on Country{name description id
                    }...on Region{name description id
                    }...on Malware{name description killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on MalwareAnalysis{result_name id
                    }...on ThreatActor{__isThreatActor:__typename name description
                    }...on Tool{name description killChainPhases{id phase_name x_opencti_order
                        }id
                    }...on Vulnerability{name description id
                    }...on Incident{name description id
                    }...on StixCyberObservable{__isStixCyberObservable:__typename x_opencti_description observable_value
                    }...on Event{name description id
                    }...on Channel{name description id
                    }...on Narrative{name description id
                    }...on Language{name id
                    }...on DataComponent{name id
                    }...on DataSource{name id
                    }...on Case{__isCase:__typename name
                    }...on Report{name id
                    }...on Grouping{name id
                    }...on ObservedData{name id
                    }...on Creator{name id
                    }...on MarkingDefinition{definition id
                    }...on ExternalReference{source_name url description id
                    }...on Artifact{id
                    }...on AutonomousSystem{id
                    }...on BankAccount{id
                    }...on CaseIncident{id
                    }...on CaseRfi{id
                    }...on CaseRft{id
                    }...on CaseTemplate{id
                    }...on CryptocurrencyWallet{id
                    }...on CryptographicKey{id
                    }...on CsvMapper{id
                    }...on Directory{id
                    }...on DomainName{id
                    }...on EmailAddr{id
                    }...on EmailMessage{id
                    }...on EmailMimePartType{id
                    }...on EntitySetting{id
                    }...on Feedback{id
                    }...on Group{id
                    }...on Hostname{id
                    }...on IPv4Addr{id
                    }...on IPv6Addr{id
                    }...on KillChainPhase{id
                    }...on Label{id
                    }...on MacAddr{id
                    }...on ManagerConfiguration{id
                    }...on MediaContent{id
                    }...on Mutex{id
                    }...on NetworkTraffic{id
                    }...on PaymentCard{id
                    }...on PhoneNumber{id
                    }...on Process{id
                    }...on PublicDashboard{id
                    }...on Software{id
                    }...on Status{id
                    }...on StixCoreRelationship{id
                    }...on StixFile{id
                    }...on StixRefRelationship{id
                    }...on StixSightingRelationship{id
                    }...on Task{id
                    }...on Text{id
                    }...on ThreatActorGroup{id
                    }...on ThreatActorIndividual{id
                    }...on Url{id
                    }...on UserAccount{id
                    }...on UserAgent{id
                    }...on WindowsRegistryKey{id
                    }...on WindowsRegistryValueType{id
                    }...on Workspace{id
                    }...on X509Certificate{id
                    }
                }
            }
        }
    }
} size=403 timestamp=2024-03-25T16: 19: 31.368Z type=READ_ERROR user={
    "group_ids": [
        "2dfd32a4-0332-4ea1-a741-b480a1c7152f",
        "0f9f0400-d10a-475e-a22d-6c670ffc44ed",
        "9176b865-df3b-48b0-ab52-fac2f34f6e18"
    ],
    "ip": "-",
    "organization_ids": [],
    "referer": "http://-:8080/dashboard/arsenal/malwares/2fd930a3-a201-4506-8d8c-b1fcbfe31b60/knowledge/overview",
    "socket": "query",
    "user_id": "88ec0c6a-13ce-5e39-b486-354fe4a7084f",
    "user_metadata": {}
} variables={
    "elementWithTargetTypes": null,
    "filters": {
        "filterGroups": [],
        "filters": [
            {
                "key": "relationship_type",
                "mode": "or",
                "operator": "eq",
                "values": [
                    "stix-core-relationship",
                    "stix-sighting-relationship"
                ]
            },
            {
                "key": "fromOrToId",
                "mode": "or",
                "operator": "eq",
                "values": [
                    "2fd930a3-a201-4506-8d8c-b1fcbfe31b60"
                ]
            }
        ],
        "mode": "and"
    },
    "first": 500,
    "fromOrToId": null,
    "orderBy": "created_at",
    "orderMode": "desc",
    "relationship_type": null
} version=6.0.7

[jborozco]

I can't reproduce the issue.

@FIying-Scotsman I noticed that you sent us only 2 out of the 3 PDF, the last one is actually a URL that I converted into a PDF and imported. Doing this, I was able to delete the 2 observables in their workbench without any error.

Do you have any other information that could help us reproduce ?

[FIying-Scotsman]

I can't reproduce the issue.

@FIying-Scotsman I noticed that you sent us only 2 out of the 3 PDF, the last one is actually a URL that I converted into a PDF and imported. Doing this, I was able to delete the 2 observables in their workbench without any error.

Do you have any other information that could help us reproduce ?

Apologies - the last URL was saved as a PDF and imported like the rest.

I made a new profile for the malware with near identical fields (description, and re-imported the same PDFs and for some reason it's no longer crashing the platform.

I then went back to the original profile and removed the single "Kill chain phase" associated and it's no longer crashing (this was the only field missing on the 2nd profile). Trying to view Attack Patterns now causes the platform to fail to load data so I believe the Kill chain Phase was the issue.

Quick testing shows all kill chain phases appear to crash the platform like before (DISARM, Mitre ATT&CK and Mitre-ics-att&ck). I've also found Attack Patterns and other pages which use ATT&CK (like tactic view on reports) fail to load on profiles and could be related.

[jborozco]

I can't reproduce either, it works well on our demo environment

[jborozco]

Since we can't reproduce, I'm closing the ticket, do not hesitate to reopen it if you have new information