Can't handle empty pgbouncer logs in incremental mode

[spookypeanut]

Caveat: I'm a bit of a pgbadger n00b. Please be gentle with me if I've missed something obvious. I have a test server (which collects logs via syslog) which sometimes has a day go by with no connections. Thus, the only thing in there are stats and one SIGHUP, eg:

May 19 05:11:01 hostname pgbouncer[7360]: stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us
May 19 05:11:01 hostname pgbouncer[7360]: got SIGHUP, re-reading config
May 19 05:12:01 hostname pgbouncer[7360]: stats: 0 xacts/s, 0 queries/s, in 0 B/s, out 0 B/s, xact 0 us, query 0 us, wait 0 us

If I use a standard incremental command line: /srv/pgbadger/pgbadger-11.8/pgbadger --incremental emptylogs/pgbouncer.log -O $PWD/emptyoutput/ I get:

FATAL: unable to detect log file format from emptylogs/pgbouncer.log, please use -f option.
    - Error at line 17888

The problem is that I want to run pgbadger on both postgres and pgbouncer logs, so I can't use -f pgbouncer. Things I've tried:

1. Running pgbadger twice, on the postgres / pgbouncer logs separately (with -f)
2. Using -f syslog - that seems to assume it's a postgres syslog file Number 1 seems to work... sometimes. If I run it on the pgbouncer log first, it does what I'd expect (this is on logs that do have queries in, btw) (-q for brevity):

   $ /srv/pgbadger/pgbadger-11.8/pgbadger -q --incremental fulllogs/postgres.log -O $PWD/fulloutput/
   $ cat fulloutput/LAST_PARSED 
   2022-05-19 09:06:26     777423  May 19 09:06:26 hostname postgres[redacted]
   pgbouncer       2022-05-19 09:06:26     0       May 19 09:06:26 hostname pgbouncer[redacted]
   $

But if I run it on postgres first, it seems to ignore the pgbouncer logs (there's no output even without -q):

$ /srv/pgbadger/pgbadger-11.8/pgbadger -q --incremental fulllogs/pgbouncer.log -O $PWD/fulloutput/
$ cat fulloutput/LAST_PARSED 
2022-05-19 09:06:26     777423  May 19 09:06:26 hostname postgres[redacted]
$

The only sure-fire way I can see at the moment to process all the logs is to have two separate output directories for pgbouncer and postgres. This would give me all the info I need, but isn't quite as convenient. Please let me know if I've missed something glaringly obvious!

[darold]

Syntax: `logfilename:format' can be used when there is multiple log format to process and they are not autodetected.

In your can use the following syntax:

/srv/pgbadger/pgbadger-11.8/pgbadger -q --incremental -O $PWD/fulloutput/   fulllogs/postgres.log emptylogs/pgbouncer.log:pgbouncer

[spookypeanut]

PERFECT! I knew you would have thought of everything :smile: I'll leave this open in case you want to investigate anything else in it, but feel free to close it from my point of view. Thank you!

[spookypeanut]

Damn. It worked fine in my test setup, but in the actual setup I'm getting the files from a remote server:

/srv/pgbadger/pgbadger-11.8/pgbadger --remote-host remotehost /srv/logdir/pgbouncer\* /srv/logdir/postgres\* --incremental --html-outdir /tmp/testhtml/ --outdir /tmp/testout/

If I add :pgbouncer on the end, I get:

Unmatched '.
Unmatched '.
FATAL: can't get size of remote file, please check what's going wrong with command:  "ls -l '/srv/logdir/pgbouncer.log
'"
    - Error at line 18406

It does the same even if I remove the wildcards:

$ /srv/pgbadger/pgbadger-11.8/pgbadger --remote-host remotehost /srv/logdir/pgbouncer.log:pgbouncer /srv/logdir/postgres.log --incremental --html-outdir /tmp/testhtml/ --outdir /tmp/testout/
Unmatched '.
Unmatched '.
FATAL: can't get size of remote file, please check what's going wrong with command:  "ls -l '/srv/logdir/pgbouncer.log
'"
    - Error at line 18406

Seems there's a newline in there, somehow. This could well be a tcsh problem (this user, sadly, uses tcsh and I can't change that). I'm just guessing from the "Unmatched" lines. If it is, I fully understand you walking away from it: tcsh is, after all, not used many places anymore.

[spookypeanut]

I think I've found the problem (forgive me, perl isn't my strongest suit). You chomp the @given_log_files, but if it has a :pgbouncer on the end, the newline is in the middle. I'll see if I can get a fix together and put in a pull request later.

[darold]

Right got it, commit b27e593 might solve this issue.