5.12.33 Elasticsearch Performance: CPU utiliczation 100%

lesley-tw commented 5 months ago

Prerequisites

[x] I read the Deployment and Setup section of the OpenCTI documentation as well as the Troubleshooting page and didn't find anything relevant to my problem.
[x] I went through old GitHub issues and couldn't find anything relevant
[x] I googled the issue and didn't find anything relevant

Description

We recently upgraded from version 5.3.17 to 5.12.33 and have encountered a performance issue with the new version. For instance, when accessing the same campaign attack pattern page in the knowledge tab, the number of queries sent to Elasticsearch increases dramatically, despite using the same GraphQL query as before. Specifically, the new version triggers about 5,800 queries to Elasticsearch, causing the CPU usage to spike to 100% for just one page. In contrast, the old version only generated about 45 queries for the same page. We suspect that other pages might have similar issues, as even a reduced number of users is leading to the same CPU overload.

Not sure it's the same issue with issue#5029

Environment

OS (where OpenCTI server runs): AWS EKS, 4 pods to serve this service, each pod 6 core cpu & 16 G memory
OpenCTI version: 5.12.33
OpenCTI client: frontend or python
Other environment details:

Reproducible Steps

Steps to create the smallest reproducible scenario:

Visit the same campaign, in knowledge tab, click attack pattern in the right menu.

In 5.3.17

show 896 attack patterns in this page
spend time
request payload
Elasticsearch query count: 45

In 5.12.33

show 896 attack patterns in this page
spend time
request payload
Elasticsearch query count: 5,873

Additional Information

In 5.3.17:

volume of data
Elasticsearch resource
- 4 data node (each node: 4 core CPU & 32 G memory)
- 2 master node (each node: 1 core CPU & 24 G memory)

In 5.12.33:

volume of data
Elasticsearch resource
- 10 data node (each node: 15 core CPU & 36 G memory)
- 2 master node (each node: 2 core CPU & 24 G memory)

richard-julien commented 5 months ago

Seems related, this screen need to be improve with a specific api to load the matrix.

vncloudsco commented 5 months ago

We also had the same situation where we ran out of cpu resources for Elasticsearch

trend-tanya-wang commented 5 months ago

Hi @SamuelHassine, may I follow up this issue? Will OpenCTI fix this issue or any suggestions? Thank you.

richard-julien commented 5 months ago

Yes, will be handled by https://github.com/OpenCTI-Platform/opencti/issues/6662

lesley-tw commented 5 months ago

Hi @SamuelHassine We are unsure if other pages have the same issue. Could you help check them? Sometimes just reading a campaign with many relationships or listing reports might also cause elasticsearch CPU issues too.

richard-julien commented 5 months ago

if you have issue when other screen please create a specific ticket describing it. Thanks

OpenCTI-Platform / opencti