search

OpenCTI-Platform / opencti

Open Cyber Threat Intelligence Platform
https://opencti.io
Other
6.33k stars 932 forks source link

Slow ingestion #1360

Closed koalaeagle closed 2 years ago

koalaeagle commented 3 years ago

Description

I am attempting to get started with OpenCTI version 4.5.2. When importing data from Alienvault OTX, I am seeing a processing rate of 3 messages per second. This will take a significant amount of time to import 2 million messages. Additionally, the rate seems to eventually drop down to 0.2 messages / second. I get similar results when disabling Alienvault and importing from a local MISP instance.

I had to increase my server to 16 vCPUs and 32GB of RAM to prevent it from running out of resources. To me it seems like the system needs a huge amount of resources and is still extremely slow to import any data (it would be months before the system had decent threat intelligence).

For now I just wanted to check in and see if anyone else has experienced similar issues or is it just me? Since it is a docker-based configuration and I have not changed anything from the example (aside from adding more feeds) I would assume my environment/performance is similar to others.

If this issue is just on my end, I can provide some more information.

Environment

  1. OS (where OpenCTI server runs): Ubuntu 20.04
  2. OpenCTI version: OpenCTI 4.5.2 (in docker, following the example)
  3. OpenCTI client: N/A
  4. Other environment details:
andrewnorthern commented 3 years ago

Hello, I am experiencing the same issue as well. I have increased my worker pool to 20 works and it has not helped.

SamuelHassine commented 3 years ago

Hello @koalaeagle and @andrewnorthern,

I think it's better to discuss that on Slack, since OpenCTI ingestion performances is something we need to qualify (depending on use cases, environments, etc.).

Please reach @richard-julien or myself directly on Slack. In the mean time, if you has not read this yet, I advise you to read carefully this one: https://medium.com/luatix/opencti-platform-performances-e3431b03f822.

Kind regards, Samuel

SamuelHassine commented 2 years ago

Close this due to inactivity.