OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.16k stars 911 forks source link

Creating a lot of relationships at once causes a timeout error #6361

Open Lhorus6 opened 6 months ago

Lhorus6 commented 6 months ago

Description

When we launch the creation of a large number of relationships in a knowledge graph, the creation process takes too long and a timeout error occurs. Here's an example of a use case where this happens: I have 350 IOCs in a report and one malware. I want to link all the IOCs with my malware, but the error occurs.

Environment

OCTI 6.0.6

Reproducible Steps

Steps to create the smallest reproducible scenario:

  1. Create a repot
  2. Go in the entities tab
  3. Add one malware
  4. Add 400 Indicators
  5. Go in the knowledge graph
  6. Try to create a relation between all indicators and the malware (in one run)

Screenshots

image

SamuelHassine commented 6 months ago

I confirm this has been reported several times. This is something we should address in the backend.

Current situation:

Whished situation:

lndrtrbn commented 5 months ago

The frontend send all relationships to create all things at once.

Does this kind of API already exists and it's just frontend code to change? Or do we also need to create this batch relationships creation API?

nino-filigran commented 5 months ago

@SouadHadjiat @aHenryJard @labo-flg Do you have any idea about the above question? ☝️

SamuelHassine commented 5 months ago

We need a new API "stixCoreRelationshipsAdd" (plurial) in addition to "stixCoreRelationshipAdd" to batch creation of relationships.

richard-julien commented 4 months ago

This one is really complex to achieve. Please come talk to me before starting it

SamuelHassine commented 1 month ago

Assigning to Julien since we are working on removing locks on relationships nodes.

SamuelHassine commented 1 month ago

Will be resolved by #6361.