Reading from multiple logs using zcat disables realtime html.

[Fieel]

Hello.

I setted up a realtime html page in which i want to display all the data from all the access logs created by nginx (not just the latest file.)

I run a simple script that runs this command at startup using crontables so the daemon is always running and i can just open the page and check the data.

At the moment i'm using this command to read only data from the access.log file: sudo goaccess -f /var/log/nginx/access.log -o /var/www/nginx_monitor/index.html --log-format=COMBINED --real-time-html --ws-url=ws://*****IP*****:7890 --ignore-crawlers --daemonize

This is what i wanted to run, the main difference is the capability to read from all logs matching the [access.log] pattern so all access.log files, even access.log.1 or access.log.2.gz get readen: `sudo zcat -f /var/log/nginx/access.log | sudo goaccess -o /var/www/nginx_monitor/index.html --log-format=COMBINED --real-time-html --ws-url=ws://IP:7890 --ignore-crawlers --daemonize`

I have several problems with the second script, data isn't even read unless i run the command again, and it won't update in real time. Why?

[allinurl]

zcat -f won't keep the pipe open, you probably can do this:

sudo zcat -f /var/log/nginx/access.log.* | sudo goaccess /var/log/nginx/access.log -o /var/www/nginx_monitor/index.html --log-format=COMBINED --real-time-html --ws-url=ws://*****IP*****:7890 --ignore-crawlers --daemonize -

Let me know if that does what you are looking for.

[Fieel]

Yes, now the command works as intended!

I still have a big problem though:

1. If I run sudo zcat -f /var/log/nginx/access.log.* | sudo goaccess /var/log/nginx/access.log -o /var/www/nginx_monitor/index.html --log-format=COMBINED --real-time-html --ws-url=ws://*****IP*****:7890 --ignore-crawlers --daemonize - at startup, the real-time page shows only the access.log data (i can see just today's data).

2. If I run the same exact command without daemonizing, it works. It reads all the data (all data since I installed nginx).

3. If I cancel and retry running the command daemonizing; it successfully reads all data.

Basically, it fails to read all the data at startup.

PS. isn't the command missing the access.log.1 file data? zcat -f /var/log/nginx/access.log.* will read all .gz files and exclude them but will exclude the access.log.1 file because isn't compressed goaccess /var/log/nginx/access.log will only read the access.log file

[allinurl]

@Fieel That's correct. GoAccess redirects file descriptors 0,1,2 to /dev/null so that the daemon can detach successfully from the tty it was started from and also so that the daemon won't write to the tty when its running.

You can try using systemd if you must. Otherwise, you can simply read historical logs and put them in one temp log file and then use goaccess to read from it. e.g.,

sudo goaccess /tmp/historical.log /var/log/nginx/access.log -o /var/www/nginx_monitor/index.html --log-format=COMBINED --real-time-html --ws-url=ws://*****IP*****:7890 --ignore-crawlers --daemonize -

[Fieel]

How come? The whole point of sudo zcat -f /var/log/nginx/access.log.* is to be able to visualize all data.

I'd rather avoid using a custom combined log if possible, isn't there a way to load all data on startup with a single command or a combination?

[allinurl]

You can certainly load and visualize all data, but if you want to daemonize the process and pipe data to it, then it will close stdin and thus you only see data from /var/log/nginx/access.log. --daemonize works best if you want to visualize the current log and not necessarily historical data.

As I mentioned before, you can try using systemd instead of --daemonize.

[Fieel]

Oookay! Apparently i misunderstood how --daemonize works, thank you for the clarification.

I want an always-running daemon feeding data to the html page, reading from all the logs available, it seems achievable, i'm so near.

I have no idea how to implement systemd in the command, what do you mean exactly?

I was thinking about using two commands, one to load all data without -daemonize (but I have no idea how to implement the 'ctrl-c' simulation to stop reading data) and then the command with --daemonize which will upload from access.log.

[allinurl]

Here's an example of systemd, but let me take a look at this and see what I find. I'll post back as soon as I have some news.

[Fieel]

That's what i was looking for, creating a service is perfect in this case. I still need the 'perfect command' to run, though!

[bitfield]

I found that while my 'always-running GoAccess script' worked perfectly when run from the command line, it would not behave properly when run as a systemd service. Although it would start with no problems, and strace confirmed that it was continuously reading logs, it never created or updated the real-time HTML output file.

I wondered if this was because GoAccess expected to be attached to a TTY, and after reading this forum post: Auto starting programs under specific tty I tried the following systemd config:

[Unit]
Description=GoAccess
After=network.target

[Service]
Type=simple
User=root
Group=root
Restart=always
ExecStart=/usr/local/bin/goaccesstail
StandardInput=tty
StandardOutput=tty
StandardError=tty
TTYPath=/dev/tty7

[Install]
WantedBy=multi-user.target

Note that I am attaching StandardInput, StandardOutput, and StandardError to tty, and setting TTYPath to a random physical TTY device (there's probably a better solution for this). Without these TTY settings, GoAccess seems unable to write to its output file.

Here is my goaccesstail script:

#!/bin/bash
/usr/bin/tail -Fq /var/log/httpd/*-access.log |grep -v '^[0-9]' |goaccess - -o /var/www/sites/default/goaccess.html --real-time-html --keep-db-files --load-from-disk --ignore-crawlers

(Note the use of tail -F, not tail -f, to keep tailing even when the log files are rotated nightly.)

The grep -v stage is to exclude log lines with the wrong format; there are multiple vhosts on each server, so I need to define a log format which includes the vhost. Older logs, from before I changed the Apache log format, are still present on disk, so I need to exclude them with this grep.

[allinurl]

@bitfield Thanks a lot for sharing the script and those findings.

[FranklinYu]

Many logs are rotated with Gzip; consider supporting this natively?

[unmec]

I was setting up a cron job to send out periodic, consolidated report, and the script failed silently. Then I found out that goaccess didn't generate required html file with zcat pipe in script when ran by cron. although the same script runs just fine when called manually.

I ended up output all log files to a temp file then ask goaccess to generate report from that file, and clean temp file when it's done. This time the cron job runs successfully.

[allinurl]

@unmec did you pass the - to the command? e.g.,

/bin/zcat access.log.*.gz | /usr/bin/goaccess access.log -

[unmec]

@allinurl Um... I don't think so.

I was doing something like this but not working:

#!/bin/sh
store_path=/somepath/hourly/
sudo zcat -f /var/log/nginx/access.log* | sudo goaccess -o $store_path`date +%F`.html --log-format='%h %^[%d:%t %^] \"%v\" \"%r\" %s %b \"%R\" \"%u\"' --date-format=%d/%b/%Y --time-format=%T

Then with this the cron runs just fine:

#!/bin/sh
store_path=/somepath/hourly/
sudo zcat -f /var/log/nginx/access.log* > $store_path`date +%F`.log
goaccess -f $store_path`date +%F`.log -o $store_path`date +%F`.html --log-format='%h %^[%d:%t %^] \"%v\" \"%r\" %s %b \"%R\" \"%u\"' --date-format=%d/%b/%Y --time-format=%T

What is the meaning of - here that could be causing this?

[allinurl]

@unmec the single dash is appended to the command line to let GoAccess know that it should read from the pipe.

[unmec]

@allinurl Thanks for the explanation, I am wiser now! ;-)