domainaware / parsedmarc

A Python package and CLI for parsing aggregate and forensic DMARC reports
https://domainaware.github.io/parsedmarc/
Apache License 2.0
961 stars 209 forks source link

High memory consumption problem #533

Open GhostRock37 opened 4 days ago

GhostRock37 commented 4 days ago

I noticed a memory consumption problem to process a large volume of emails (98000). I tried to process the mails with a freshly installed Azure machine of 32 GB of RAM and after a few hours of processing, all the RAM was used and the processing failed (elasticsearch KO). I'm on the last parsedmarc version 8.12.

I have an ELK instance installed with parsedmarc on this machine.

image001.png

By controlling the memory consumption of the Parsemarc process (Command Top on Linux) I notice that over time, the process consumes more and more memory ... inexorably

Here is a graph extracted from Azure monitoring for the machine concerned

parsedmarc conso3.jpg