search

darold / pgbadger

A fast PostgreSQL Log Analyzer
http://pgbadger.darold.net/
PostgreSQL License
3.57k stars 350 forks source link

Large files are ignored #441

Closed eazrael closed 6 years ago

eazrael commented 6 years ago

I have a log larger than 2GB and it is ignored by pgbadger. Compressed with xz pgbadger accepts it:

# ls -alh /var/log/postgres
total 4.6G
drwxr-xr-x  2 postgres root   147 Sep 17 00:22 ./
drwxr-xr-x 13 root     root  4.0K Sep 16 04:40 ../
-rw-rw----  1 postgres users 3.1M Sep 14 23:59 postgresql-2018-09-14.log.xz
-rw-rw----  1 postgres users 3.9M Sep 15 23:54 postgresql-2018-09-15.log.xz
-rw-rw----  1 postgres users  43M Sep 16 23:59 postgresql-2018-09-16.log.xz
-rw-rw----  1 postgres users 2.5G Sep 17 21:30 postgresql-2018-09-17.log
# /usr/local/bin/pgbadger -o xxx.html -O /zzz/yyy/xxx/ -Z +2 -I /var/log/postgres/*.{log,log.xz} -j 8 -d xxx --exclude-app=pg_dump --exclude-query="^(COMMIT|ALTER)" --start-monday
[========================>] Parsed 128863972 bytes of 128863972 (100.00%), queries: 0, events: 0
[========================>] Parsed 173105090 bytes of 173105090 (100.00%), queries: 0, events: 0
[========================>] Parsed 853983809 bytes of 853983809 (100.00%), queries: 0, events: 0
LOG: Ok, generating global index to access incremental reports...
# time nice -n 19 /usr/local/bin/pgbadger -o xxx.html -O /zzz/yyy/xxx/ -Z +2 -I /var/log/postgres/*.log -j 8 -d xxx --exclude-app=pg_dump --exclude-query="^(COMMIT|ALTER)" --start-monday
LOG: Ok, generating global index to access incremental reports...
# xz -T8 -3vv < /var/log/postgres/postgresql-2018-09-17.log > /tmp/log.xz
xz: Filter chain: --lzma2=dict=4MiB,lc=3,lp=0,pb=2,mode=fast,nice=273,mf=hc4,depth=48
xz: Using up to 8 threads.
xz: 537 MiB of memory is required. The limiter is disabled.
xz: Decompression will need 5 MiB of memory.
  100 %     166.8 MiB / 2,525.1 MiB = 0.066    73 MiB/s       0:34 
# /usr/local/bin/pgbadger -o xxx.html -O /zzz/yyy/xxx/ -Z +2 -I /tmp/log.xz -j 8 -d xxx --exclude-app=pg_dump --exclude-query="^(COMMIT|ALTER)" --start-monday
[========================>] Parsed 2647764275 bytes of 2647764275 (100.00%), queries: 915453, events: 9
LOG: Ok, generating HTML daily report into /zzz/yyy/xxx//2018/09/17/...
LOG: Ok, generating HTML weekly report into /zzz/yyy/xxx//2018/week-39/...
LOG: Ok, generating global index to access incremental reports...

pgbadger version 10.1

darold commented 6 years ago 
$ ls -lah /data/test/huge_pg_logs/

drwxr-xr-x 2 gilles gilles 4,0K oct.  17 18:58 .
drwxr-xr-x 5 gilles gilles 4,0K sept. 12 11:58 ..
-rw-r--r-- 1 gilles gilles 1,9G sept. 12 12:10 postgresql-Fri.log
-rw-r--r-- 1 gilles gilles 543M oct.  17 18:49 postgresql-Sat.log
-rw-r--r-- 1 gilles gilles 632M sept. 12 12:10 postgresql-Sun.log
-rw-r--r-- 1 gilles gilles 3,2G sept. 12 12:15 postgresql-Thu.log
-rw-r--r-- 1 gilles gilles 3,3G sept. 12 12:03 postgresql-Tue.log
$ perl pgbadger -j 12 /data/pglog/huge_log_test/postgresql-*
[========================>] Parsed 10031788505 bytes of 10031788505 (100.00%), queries: 60366965, events: 3965
LOG: Ok, generating html report...

Everything is working fine with all files.

eazrael commented 6 years ago

Can't reproduce it, too. At that time it was reproducible and I used the same files now. Maybe some glitch in the logs.