Incremental reports not working in 11.6 or latest

[hansgv]

After upgrading to 11.6, HTML files no longer get created in my outdir. I noticed someone reported a similar issue #692 which is marked as fixed. I downloaded and installed the latest code but the issue persists.

pgbadger --quiet --dns-resolv --iso-week-number --format rds --dbname mydb --jobs 2 --incremental --retention 2 --extra-files --exclude-user exludeduser /logs/error/postgresql.log* --outdir /var/www/mysite/pgbadger/

I'm on Ubuntu 20.04. Installed manually from source.

[dudleyperkins]

yes i marked it as solved - i just grabbed the version darold linked to, that had the commit in it and it worked like a charm.

did you test the latest release (it's not fixed there yet) or did you test it with the current pgbadger file from the code page? Latter would mean you might hit a new bug

[hansgv]

Yes, I downloaded and installed the latest code. Even double-checked that commit https://github.com/darold/pgbadger/commit/489ac52194e002c2373e24eb9af1a84bfb7214b3 was actually in it.

[darold]

I can not reproduce the issue, I use the same command except the log format. Can you verify that pgbadger is able to build a report for one of you log without using incremental mode?

[hansgv]

@darold With version 11.5:

pgbadger --dns-resolv --format rds --dbname mydb --jobs 2 --incremental --retention 2 --extra-files logs/error/postgresql.log* --outdir pgbadger/test/

...generates global/weekly/daily HTML files under the pgbadger/test/ directory. Version 11.6 doesn't. Both versions generate the .bin files.

If I drop the --incremental option, both versions generate an out.html file under pgbadger/ directory.

[baldpope]

I think I'm running into this same problem, this appears to be affecting us after we upgraded from postgres 9.6.17 to 12.7 this weekend - since the upgrade none of the reports are building in the output directory specified. we are using aurora postgresql, and I noticed above some people are using rds as the logtype, where I am (and have been) using stderr, not sure if that's relevant.

here is our invocation:

pgbadger -f stderr -p '%t:%r:%u@%d:[%p]:' -X -I -s 10 -t 20 -J 20 -j 20find ${LOGDIR} -size +1c -type f -name "postgresql.log*"-R 8 -O /mnt/aws/reports

prior to this weekend, pgbadger successfully created the daily/weekly reports. I notice that in my example, I'm NOT passing in the dbname - should I?

[darold]

It definitively works at my side, first run:

$ perl pgbadger --dns-resolv --format stderr --dbname valbois --jobs 2 --incremental --retention 2 --extra-files /opt/data_files/multi-small-files/postgresql-2016-12-31_* -J 8 --outdir /home/git/pgbadger/report/
[========================>] Parsed 668619016 bytes of 668619016 (100.00%), queries: 2592029, events: 543
LOG: Ok, generating HTML daily report into /home/git/pgbadger/report//2016/12/31/...
LOG: Ok, generating HTML weekly report into /home/git/pgbadger/report//2016/week-53/...
LOG: Ok, generating global index to access incremental reports...

$ tree /home/git/pgbadger/report/
/home/git/pgbadger/report/
├── 11
│   ├── bean.min.js
│   └── [...]
├── 2016
│   ├── 12
│   │   └── 31
│   │       ├── 2016-12-31-28583.bin
│   │       ├── [...]
│   │       └── index.html
│   └── week-53
│       └── index.html
├── index.html
└── LAST_PARSED

second run:

$ perl pgbadger --dns-resolv --format stderr --dbname valbois --jobs 2 --incremental --retention 2 --extra-files /opt/data_files/multi-small-files/postgresql-2017-01-01_* -J 8 --outdir /home/git/pgbadger/report/
[========================>] Parsed 860007986 bytes of 860007986 (100.00%), queries: 1657019, events: 557
LOG: Ok, generating HTML daily report into /home/git/pgbadger/report//2017/01/01/...
LOG: Ok, generating HTML weekly report into /home/git/pgbadger/report//2017/week-02/...
LOG: Ok, generating global index to access incremental reports...

$ tree /home/git/pgbadger/report/
/home/git/pgbadger/report/
├── 11
│   ├── bean.min.js
│   └── [...]
├── 2016
│   ├── 12
│   │   └── 31
│   │       ├── 2016-12-31-28583.bin
│   │       ├── [...]
│   │       └── index.html
│   └── week-53
│       └── index.html
├── 2017
│   ├── 01
│   │   └── 01
│   │       ├── 2017-01-01-28744.bin
│   │       ├── [...]
│   │       └── index.html
│   └── week-02
│       └── index.html
├── index.html
└── LAST_PARSED

perhaps your log file format has changed. Closing.

[hansgv]

@darold I still experience this issue. What additional information can I provide to help resolve it?

[darold]

With latest development code too?

[hansgv]

@darold Yes.

[martonsz]

I am having the same issue. I had to downgrade to 11.5 from 11.6 to make it work again with the --incremental flag.

I haven't tried with the development code. I am downloading the source code from the release page on github.

[darold]

I have fixed an issue in commit 202050f related to the use of --iso-week-number in incremental mode but it is probably not related to the same problem. If one of you can share with me on my private email several small logs to use in incremental mode that reproduce the issue I will be able to find the solution but on my tests everything happens as expected.

[martonsz]

@darold I think I found something!

When I was creating a sample log file for you I changed the database name in the logs to make it less sensitive for the public. But instead of changing the parameter --dbname for pgbadger when making a test run. I just removed it. And that made it work! It is generating the index.html.

So I tried this with my original log file and removed the --dbname. Now v11.6 works with the --incremental flag.

But now I am having unwanted databases in my report :(

[darold]

You must use latest development code this issue has been fixed by commit 489ac52.

[martonsz]

@darold I missed that issue about --dbname.

I have managed to run --incremental with the latest commit 202050fc6b0505e37c6be6360be17a7efd7a1e03 and it works fine. Looking forward to the next release so I can use it in production.

[hansgv]

@darold --incremental still doesn't work for me event after the latest 202050f commit.

Please allow me to clarify my setup.

I store postgres log files in 1 hour increments and process them hourly within the next hour. When starting from scratch (i.e. a bunch of postgres log files are available but pgbadger output directory is empty) the initial pgbadger execution runs as expected. It generates the output directory structure, LAST_PARSED, .bin files, and HTML files.

However, subsequent incremental pgbadger runs generate new hourly .bin files properly but don't generate HTML files. My assumption is that the hourly runs should generate/re-generate the affected HTML files (typically the ongoing day's).

Here's an example of what today's directory currently looks like. As you can see, a bunch of .bin files have been generated hourly but no HTML file for the day. One wasn't even generated when the day directory was generated.

ls -lh /var/www/mysite/pgbadger/2021/12/07/
-rw-r--r-- 1 root root 126K Dec  7 03:35 2021-12-07-318584.bin
-rw-r--r-- 1 root root 145K Dec  7 04:35 2021-12-07-341717.bin
-rw-r--r-- 1 root root 122K Dec  7 05:35 2021-12-07-364620.bin
-rw-r--r-- 1 root root 144K Dec  7 06:35 2021-12-07-388272.bin
-rw-r--r-- 1 root root 184K Dec  7 07:35 2021-12-07-411441.bin
-rw-r--r-- 1 root root 180K Dec  7 08:35 2021-12-07-435710.bin
-rw-r--r-- 1 root root 195K Dec  7 09:35 2021-12-07-459905.bin
-rw-r--r-- 1 root root 210K Dec  7 10:35 2021-12-07-483298.bin
-rw-r--r-- 1 root root 205K Dec  7 11:35 2021-12-07-506612.bin
-rw-r--r-- 1 root root 224K Dec  7 12:35 2021-12-07-530036.bin
-rw-r--r-- 1 root root 214K Dec  7 13:35 2021-12-07-553382.bin
-rw-r--r-- 1 root root 204K Dec  7 14:35 2021-12-07-576684.bin
-rw-r--r-- 1 root root 202K Dec  7 15:35 2021-12-07-601060.bin
-rw-r--r-- 1 root root 219K Dec  7 16:35 2021-12-07-624473.bin
-rw-r--r-- 1 root root 238K Dec  7 17:35 2021-12-07-648023.bin
-rw-r--r-- 1 root root 238K Dec  7 18:35 2021-12-07-672115.bin
-rw-r--r-- 1 root root 241K Dec  7 19:35 2021-12-07-696011.bin

[darold]

what is your exact pgbadger or command cron entry?

[hansgv]

@darold

ionice -c3 pgbadger \
--dbname k2 \
--jobs 2 \
--format rds \
--iso-week-number \
--dns-resolv \
--incremental \
--extra-files \
--exclude-user admin_one \
--exclude-user admin_two \
--exclude-user admin_three \
--outdir '/var/www/mysite/pgbadger' \
--html-outdir '/var/www/mysite/pgbadger' \
/hourly-logs/postgresql.log*