Heavy slowdown when executing script

[Parasdeepkohli]

Hi,

I was successfully able to configure the application on the same Ubuntu 18.04 server as my MISP instance. But when I try to push ~110k indicators to Microsoft Graph Security, the script always slows down halfway through.

For example, from 0 to ~50k indicators, the script takes 6-8 seconds per 100 indicators. But after that, it takes over 500 seconds per 100 indicators. Is there any way to diagnose the cause of this slowdown, and if so, can it be fixed?

[cudeso]

There are a couple of things to look into

• When you execute the script, do a top and check the system/memory usage
• Do you have OOM messages on your system?
• Are there specific error log entries (the JSON output in the logs/ directory (defined in LOG_DIRECTORY_NAME))
• Is there an application gateway / proxy in between that can be responsible for some form of rate limiting?

[Parasdeepkohli]

Hi Cudesco,

Sorry for the late reply. Swamped with work and no time to do some testing. Okay, so I tried troubleshooting the first 3 points you mentioned and I observed the following:

• Even after the script had slowed down to 50s per 100 indicators, CPU and Memory Utilization in top was fine. It hovered around 12.5% CPU and 2.6% Memory.
• I did not find any OOM messages on the system
• I found 3 error log entries mid-way through execution, and each of them was a 'Target: body - ValidationFailed - Validation of indicator content failed' error. They seemed to have popped up at random intervals. Can't find any pattern there.

Looks like I will have to check whether a gateway / proxy is causing rate limiting. While writing this comment, the script is trying to push it's largest batch of indicators yet: 360,000+. It has successfully processed about 170,000 of them and is chugging along at a speed of 100 indicators per 60 seconds. It started off nice and zippy at 1.5s though. Strange behavior. Will keep you posted!

[cudeso]

Can you check with the new Upload Indicators API? The integration takes into account the throttling set by Microsoft (100indicators/request; 100 requests per minute). Latest commits include the documentation and fixes.

[cudeso]

Closing, pending feedback