search

MISP / MISP

MISP (core software) - Open Source Threat Intelligence and Sharing Platform
https://www.misp-project.org/
GNU Affero General Public License v3.0
5.38k stars 1.4k forks source link

PHP memory limit suggestion > 13GB! #7572

Open chrisnewmanuk opened 3 years ago

chrisnewmanuk commented 3 years ago

Work environment

Questions Answers
Type of issue Support
OS version (server) Ubuntu 18 LTS
PHP version 7.4.14
MISP version / git hash v2.4.136

Support Questions

"Event fetch potential memory exhaustion. During the fetching of events, a large event (#2916) was detected that exceeds the available PHP memory. Consider raising the PHP max_memory setting to at least 13805M "

This event has a large number of events that are being pulled via API. The suggestion is to raise the mem limit to 13G for PHP!

This isn't viable - please advise on best steps

Logs

[Mon Jul 19 08:07:20.070741 2021] [php7:error] [pid 1326] [client 192.168.2.147:39960] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php on line 1753 [Mon Jul 19 08:34:01.662220 2021] [php7:error] [pid 1050] [client 192.168.2.193:33252] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 4096 bytes) in /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php on line 285 [Mon Jul 19 08:42:55.255079 2021] [php7:error] [pid 1049] [client 192.168.2.193:33332] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 4096 bytes) in /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/Database/Mysql.php on line 285 [Mon Jul 19 08:56:53.669911 2021] [php7:error] [pid 2570] [client 192.168.2.193:33376] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/MISP/app/Model/Event.php on line 883 [Mon Jul 19 09:14:36.631108 2021] [php7:error] [pid 2932] [client 192.168.2.193:33444] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/MISP/app/Model/Event.php on line 883 [Mon Jul 19 09:28:35.307443 2021] [php7:error] [pid 2934] [client 192.168.2.193:33496] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 20480 bytes) in /var/www/MISP/app/Model/Event.php on line 883 [Mon Jul 19 09:42:35.262054 2021] [php7:error] [pid 2934] [client 192.168.2.193:33548] PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 192512 bytes) in /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php on line 502

iglocska commented 3 years ago

That sounds like an utterly massive event. Where are you trying to fetch that from? Any chance to ask the partner to restructure that event?

chrisnewmanuk commented 3 years ago

It is > 350,.000 hash's from Malware Bazaar / Abuse.ch- a preloaded MISP feed :p

image

image

It does export successfully quite a few times but eventually, I start getting PHP memory issues - something may not be free'ing

It eventually kills the AWS instance (m5 large) and has to be recovered

adulau commented 3 years ago

Abuse.ch provides a daily feed in MISP format. It might more appropriate than a single CSV import.

chrisnewmanuk commented 3 years ago

I'll have a look - but we also have other feeds with over 250k entries so..... the bug still persists :p Ingestion isnt a problem at all and its not had a hiccup over that - its where a big export happens and it works until PHP consumes all of the RAM. Also, if the PHP session continues to consume RAM without free'ing it, may still be in the same position

cac0ns3c commented 3 years ago

Having the same issue only with almost 60Gb... any advise?

Benni0 commented 2 years ago

@adulau @iglocska

I am sorry to bring up this old problem again, but the suggestion from @adulau does not work in practice. Enabling MISP feeds from Abuse.ch are highly correlated, which results in massive relations table growth (https://github.com/MISP/MISP/issues/8275). It is also not possible to import the (full) csv files, because of increasing memory usage.

I have created a MISP feed for threatfox IOCs, where I've seperated the IOC to montly events with no duplicates, which should reduce correlations. During the synchronization of this feed, several events with <100k attributes are synchronized successfully. But at some point the memory limit is exceeded, during the synchronization of an event which also had <100k attributes. This leads me to the assumption that a memory leak exists here.

Any Ideas where this Leak could happen?