search

cudeso / misp2sentinel

MISP to Sentinel integration
MIT License
52 stars 17 forks source link

CPU Maxed when using Upload Indicators API #89

Closed Hexa60n closed 3 months ago

Hexa60n commented 3 months ago

Hi, I have been using misp2sentinel to push to the Graph API for around 12 months now (thanks for the work!). I have attempted to move to the Upload Indicators API but whenever I run the script the CPU maxes at 100%. I left it for ~90 minutes and it was still searching for attributes within the valid_from and valid_until range.

By comparison the Graph API manages to search and push around ~1million attributes in 5 minutes on the same server.

Additionally when using Graph API, CPU utilisation sits at a comfortable range for the duration.

Do you have any ideas what may be causing this?

cudeso commented 3 months ago

Hello. The biggest change between the Graph API and the Upload Indicators API is that the latter uses the misp2stix library. This is necessary because Upload Indicators API requires STIX. When fetching large sets of data, this can greatly increase the CPU/memory usage. The Graph API is/was fairly straightforward, but for Upload Indicators the translation to STIX is needed.

To counter the memory usage, limit the number of MISP events that are processed per 'batch', do this with misp_event_limit_per_page (in config.py). For example set it to 50. The downside is that the lower you set this number, the more time it will take to finish.

Hexa60n commented 3 months ago

Hi, thanks for the feedback. In my previous implementation I was scanning quite a large set of data (timestamp=30d) which worked OK with Graph API but was causing this issue. Reducing that to 24h has solved this problem with Upload Indicators API (I hadn't thought about changing it so drastically as I was still tinkering with all the other new settings). Thanks!

cudeso commented 3 months ago

Good to hear it solved your problem!