search

userjack6880 / Open-DMARC-Analyzer

Open DMARC Analyzer is an Open Source DMARC Report Analyzer to be used with DMARC reports that have been parsed by John Levine's rrdmarc script or techsneeze's dmarcts-report-parser.
GNU General Public License v3.0
227 stars 25 forks source link

[Question]: How large does this scale? #96

Open sporkman opened 1 year ago

sporkman commented 1 year ago

I'm having no issues with the other piece of this (the report parser), but this web UI is acting a bit odd, and I'm wondering if I'm just stuffing too much data into it.

For example, with the 1,000 or so reports I'm looking at, that represents about 43,882,401 messages.

Front page loads fine, but clicking in to any domain eventually either just times out or results in a connection reset.

Server is currently not doing anything else and I'm not seeing anything of note in "top" while the UI is loading. Setup is bog-standard apache with mod_php, db is postgres.

Before I go too nuts with this, just wanted to reach out and see if it's practical to be working with this many records.

userjack6880 commented 1 year ago

Honestly, I do not know how far this will scale - the goal is to scale up, but I admit there are inefficiencies in the queries that need to be worked out. Some methods to attack this are being considered.

sporkman commented 1 year ago

OK, let me know if you want a look at what I've got at some point - again, not really sure if the numbers we're looking at are medium, large, XL..

userjack6880 commented 1 year ago

Thanks for the feedback - I'm adding this to the V2 Alpha milestone.

rtm516 commented 3 months ago

It works on large domains if you are prepared to wait for the page to load Based on the metrics on the homepage over the last ~24h we have had 5k total and that took 10.9 min according to Chrome