search

darold / pgbadger

A fast PostgreSQL Log Analyzer
http://pgbadger.darold.net/
PostgreSQL License
3.49k stars 349 forks source link

Running out of memory when generating monthly reports #766

Closed jsabater closed 1 year ago

jsabater commented 1 year ago

Good day!

I am having trouble generating monthly reports in a 12 GB RAM container for the month of december of 2022, and it was a slow month. Only two things that changed from previous months were:

  1. I updated pgbadger to version 12.0.
  2. Four very small databases were added, and only one of them was actually in production.

In the end I managed to allow the job to finish by giving it 16 GB of RAM (it usually has 8 GB). The command was this:

/usr/bin/pgbadger --extra-files --month-report 2022-12 --incremental --jobs 6 --start-monday --verbose /var/www/pg_reports

I checked the documentation and there seems to be nothing to be done here, as all binary files are loaded before generating the HTML output.

Is there any way to prevent pgbadger to use so much RAM, maybe operating in a much slower manner? Also I am not sure whether using less jobs would make a difference, but I think not as per the documentation. Also running the monthly report every day or week instead of waiting for the end of the month seems that would not make a difference either, as per the documentation:

The report for a current month can be run every day, it is entirely rebuilt each time.

Any hints?

Thanks in advance.

darold commented 1 year ago

Monthly report uses lot of memory, I do not recommend it. Weekly report should be enough. Maybe I can based monthly report on weekly stats instead of daily stats but I'm not convinced that it will me more efficient.