`--real-time-html` results not updated when log is updated

[alexpreynolds]

Setup

I have a cron job that updates a summary log once per hour.

The summary is made up of a concatenation of two weeks worth of nginx logs and applies grep on the results, for a key for which we are interested in tracking statistics, e.g.:

$ grep foo <(zcat access.log.*.gz | cat access.log - ) > summary.log

The goaccess binary is compiled with debug support enabled and port 7890 is opened on the host:

# make clean
# ./configure --enable-utf8 --enable-geoip=legacy --with-openssl --enable-debug
# make && make install

We start goaccess with various parameters. The debug flag writes its log messages to standard output/error:

$ sudo goaccess summary.log -o report.html --real-time-html
Parsing... [0] [0/s]
WebSocket server ready to accept new client connections
Accepted: 13 1.2.3.4
Active: 0
Accepted: 13 1.2.3.4
Active: 0
Accepted: 13 1.2.3.4
Active: 0
Accepted: 13 1.2.3.4
Accepted: 14 1.2.3.5
Active: 1
Accepted: 13 1.2.3.4
Active: 1
Active: 0
...

The goaccess process correctly renders an analysis of the report log, as it was made initially.

Observed result

The problem is that goaccess only reports statistics for the static file summary.log when that log file was first made.

My understanding of the --real-time-html is that the goaccess report page should update itself, once summary.log is updated.

I have a cron job that runs the equivalent of the following, once per hour (paths simplified for describing this issue):

$ grep foo <(zcat access.log.*.gz | cat access.log - ) > summary.log

I can verify that the summary.log file is updated once per hour with new log hits.

The only way to get goaccess to show the analysis of this updated file is by stopping and restarting the goaccess process.

Expected result

This does not jibe with my understanding of what the --real-time-html option is supposed to do.

My expectation was that goaccess will update what it presents, as the log file is updated.

Question

How do I get goaccess to render an updated summary file, when it is updated, without stopping and restarting the goaccess process?

[alexpreynolds]

This issue may be related: https://github.com/allinurl/goaccess/issues/1803

[allinurl]

Sounds like you are rewriting the summary.log each time you run grep. Could you run it as?

# tail -f -n +0 access.log | grep --line-buffered 'foo' | goaccess -o out.html --real-time-html - 

[alexpreynolds]

My end goal is to run a daemon at startup on a collection of log files (some gzip-ed).

If goaccess can't track the summary when it changes, that's fine.

My "solution" is to run a cron job every hour to create a summary log file from the current log file and these archives, and then re-run goaccess every five minutes after the hour (without --real-time-html) to generate a static report.

I guess the issue is still that --real-time-html isn't reporting the true, current state of the log files, but maybe this and issue #1803 might be useful for testing.

[allinurl]

So far, from what I understand, it seems to me that simply re-running goaccess and generating a static report may be what you are looking for. GoAccess can keep track of previous runs by utilizing --persist and --restore. Though, the --real-time-html option was designed to open the log once and use fseek(3) and track the offset upon appending new data. e.g., tail -F

[alexpreynolds]

The cron job is effectively appending data, or at least it must be changing file offsets when the file content is changed.

Updating this statically is fine for my use case. I would perhaps suggest qualifying what this option can do, if it is only for a certain use case, or perhaps test if it works in a scenario where the input file data actually changes, as that does not appear to be the case here.

[satyapm10]

Setup

I have a cron job that updates a summary log once per hour.

The summary is made up of a concatenation of two weeks worth of nginx logs and applies grep on the results, for a key for which we are interested in tracking statistics, e.g.:

$ grep foo <(zcat access.log.*.gz | cat access.log - ) > summary.log

The goaccess binary is compiled with debug support enabled and port 7890 is opened on the host:

# make clean
# ./configure --enable-utf8 --enable-geoip=legacy --with-openssl --enable-debug
# make && make install

We start goaccess with various parameters. The debug flag writes its log messages to standard output/error:

$ sudo goaccess summary.log -o report.html --real-time-html
Parsing... [0] [0/s]
WebSocket server ready to accept new client connections
Accepted: 13 1.2.3.4
Active: 0
Accepted: 13 1.2.3.4
Active: 0
Accepted: 13 1.2.3.4
Active: 0
Accepted: 13 1.2.3.4
Accepted: 14 1.2.3.5
Active: 1
Accepted: 13 1.2.3.4
Active: 1
Active: 0
...

The goaccess process correctly renders an analysis of the report log, as it was made initially.

Observed result

The problem is that goaccess only reports statistics for the static file summary.log when that log file was first made.

My understanding of the --real-time-html is that the goaccess report page should update itself, once summary.log is updated.

I have a cron job that runs the equivalent of the following, once per hour (paths simplified for describing this issue):

$ grep foo <(zcat access.log.*.gz | cat access.log - ) > summary.log

I can verify that the summary.log file is updated once per hour with new log hits.

The only way to get goaccess to show the analysis of this updated file is by stopping and restarting the goaccess process.

Expected result

This does not jibe with my understanding of what the --real-time-html option is supposed to do.

My expectation was that goaccess will update what it presents, as the log file is updated.

Question

How do I get goaccess to render an updated summary file, when it is updated, without stopping and restarting the goaccess process?

any solutions on this?

[rusk0]

I am also wondering about that as well: GoAccess - 1.7.2.

Command :

/usr/bin/goaccess /var/log/nginx/production-access.log -o /var/www/html/production/index.html --log-format='%^ %^ %v [%d:%t %^] %~ %m %U %H %^ %^ %T %^ %s %b %R "%u" "%h" %^ %^' --time-format='%H:%M:%S' --date-format='%d/%b/%Y' --real-time-html and putting it in background

(Also tried --daemonize and the tail -f -n +0 suggested here)

I am still unable to get real time stats ?

Am i missing something ?

Going to leverage cronjobs for now as well.

[allinurl]

Hey @rusk0, are you seeing the green dot in the HTML report? It should be located near the gears icon in the navigation bar. If you're not seeing it, please check the console in your browser's developer tools.

[cppgent0]

I'm having the same problem (Ubuntu 22.04)

$ goaccess --version
GoAccess - 1.8.1.
  --enable-utf8
  --enable-geoip=mmdb
  --with-getline
  --with-openssl

Start it with:

sudo goaccess -f /var/log/nginx/access.log -o /var/www/goaccess/index.html --real-time-html --addr=localhost --port=8022

$ ss -l 2>&1 | grep 8022
tcp   LISTEN 0      4096                                    127.0.0.1:8022                     0.0.0.0:*          

The nginx conf is:

    location /goaccess/ {
        root /var/www/goaccess;

        # no cache
        expires -1;
        sendfile off;
        add_header Cache-Control no-cache;
    }

The console shows:

For a few days it was working. I had goaccess started up in a service no problem I did ... something ... and now it's not live updating. If I restart goaccess it works perfectly. the goaccess.conf seems to be correct (as far as I know).

BTW I have tried adding these --ws-url to the command line, no joy:

• --ws-url localhost:8022
• --ws-url /goaccess:8022
• --ws-url /goaccess:8022/

[allinurl]

@cppgent0, can you recall the actions you performed that led to the issue? Have you tried running it using the following command:

# goaccess /var/log/nginx/access.log -o /var/www/goaccess/index.html --real-time-html

Ensure that port 7890 is open.

[cppgent0]

can you recall the actions you performed that led to the issue?

I wish! But sorry, no.

Have you tried running it using the following command:

Just tried it now, got the same warning in the Chrome console

also tried it with the full url for --ws-url:

sudo goaccess -f /var/log/nginx/access.log -o /var/www/goaccess/index.html --real-time-html --addr=localhost --port=8022 --ws-url=arrizza.com/goaccess/

and same error.

[allinurl]

@cppgent0 Did you provide the proper certificate and key in goaccess? It appears that you're attempting to access via HTTPS, indicating a secure WebSocket (WSS) connection.

[MathNodes]

@cppgent0, can you recall the actions you performed that led to the issue? Have you tried running it using the following command:

# goaccess /var/log/nginx/access.log -o /var/www/goaccess/index.html --real-time-html

Ensure that port 7890 is open.

This was the issue for us. need to open port 7890 with ufw