Multiple Realtime HTML Reports with Multiple PIPES via SSH Fails

[gunit84]

Hi allinurl,

First of all thanks creating this software. I am having trouble getting simultaneous real time html reports via ssh with multiple pipes and I am not sure why.

I am using a compiled version of Go Access version 1.2 on RHEL 7.

Can you please shed some light on this. I have tried multiple config files and just the CLI as I am showing below. Also goaccess does not create the PIPE hence thats why I do it below.

Server 1

# Allow the Firewall
firewall-cmd --permanent --add-port=8890/tcp
firewall-cmd --reload
firewall-cmd --permanent --list-all

Create the Custom Pipes

mkfifo -m 666 /tmp/server1in.fifo && mkfifo -m 666 /tmp/server1out.fifo  

Run the command

ssh root@server1 'tail -f /var/log/httpd/access_log' | goaccess --no-global-config --date-format='%d/%b/%Y' --time-format='%H:%M:%S' --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u"' --log-file='-' --fifo-in=/tmp/server1in.fifo --fifo-out=/tmp/server1out.fifo -a --log-format=COMBINED - --real-time-html  -o /var/www/html/report_server1_webserver.html --geoip-database=/opt/Downloads/GeoIP.dat --html-report-title=server1_webserver_http_access_log  --ignore-referer=*.test.local --ws-url=mainserver --real-os -d --max-items=20 --addr=0.0.0.0 --port=8890   --daemonize -l /tmp/server1_goaccess_debug

Server 2

# Allow the Firewall
firewall-cmd --permanent --add-port=9890/tcp
firewall-cmd --reload
firewall-cmd --permanent --list-all

Create the Custom Pipes

mkfifo -m 666 /tmp/server2in.fifo && mkfifo -m 666 /tmp/server2out.fifo

Run the command

ssh root@server2 'tail -f /var/log/httpd/access_log' | goaccess --no-global-config --date-format='%d/%b/%Y' --time-format='%H:%M:%S' --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u"' --log-file='-' --fifo-in=/tmp/server2in.fifo --fifo-out=/tmp/server2out.fifo -a --log-format=COMBINED - --real-time-html  -o /var/www/html/report_server2_webserver.html --geoip-database=/opt/Downloads/GeoIP.dat --html-report-title=server2_webserver_http_access_log --ignore-referer=*.test.local --ws-url=mainserver --real-os -d --max-items=20 --addr=0.0.0.0 --port=9890   --daemonize -l /tmp/server2_goaccess_debug 

Only Server 1 works while Server 2 fails.

When I run this command it usually holds the Window then the Shell dies.

Here is the output of the debug log of Server2 (the failing one):

Opened GeoIP City database: /opt/Downloads/GeoIP.dat Bye.

So then I removed the geoip directive and all it says was. Bye.

Thanks

[allinurl]

GoAccess does create the pipe on my side. Here's what I ran and it worked fine with live stats.

ssh root@host.com 'tail -f /var/log/apache2/access.log' | ./goaccess --no-global-config --log-format=VCOMBINED - --real-time-html -o /home/user/www/site/live.html --daemonize

One thing, I noticed is that you are using --log-file, you can get rid of it. I'd recommend keeping it simple and run goaccess with the least amount of cmd options. Also, make sure you are using absolute paths if you are using --daemonize. Additionally, if you don't need --ws-url I wouldn't use it unless you are really sure about the location where the websocket server is running.

[gunit84]

Hi allinurl thanks for your reply.

These commands work but not at the SAME time.

ssh root@server1 'tail -f /var/log/httpd/access_log' | goaccess --no-global-config --log-format=COMBINED - --real-time-html -o /var/www/html/report_server1_webserver.html --fifo-in=/tmp/server1.fifo --fifo-out=/tmp/server1.fifo --daemonize

ssh root@server2 'tail -f /var/log/httpd/access_log' | goaccess --no-global-config --log-format=COMBINED - --real-time-html -o /var/www/html/report_server2_webserver.html --fifo-in=/tmp/server2.fifo --fifo-out=/tmp/server2.fifo --daemonize

With the SSH tunnels going to completely different servers and different reports, the simultaneous reports start distorting the data of both reports at the same time.

So with both reports running it's really only reporting from 1 of the tunnels. So this reports the same data on both reports simultaneously instead of data from server 1 on server 1 report and data from server 2 on server 2 report.

This is why I created separate PIPES because I was under the impression that goaccess was sharing the same PIPE and that the same data was coming in via one of the tunnels. However this makes no difference.

The more I think about it I feel it's got something to do with STDIN or that it needs it's own separate File Descriptors?? Please let me know your thoughts.

Please not this is for when you have 2 Sessions of the tunneling running with 2 different servers and 2 different reports tailing the log at the same time.

Thanks again.

[allinurl]

I see what you mean. A workaround for this would be to pipe it to a file on your local machine and then read that file with goaccess. However, let me try to reproduce this on my side and see what's going on.

[allinurl]

A follow-up on this. I was able to run the following successfully:

ssh root@server1 'tail -f /var/log/apache2/access.log' | goaccess --no-global-config --log-format=VCOMBINED --real-time-html -o /home/user/1.html --port=7890 --fifo-in=/tmp/1.in --fifo-out=/tmp/1.out --daemonize

AND

ssh root@server2 'tail -f /var/log/apache2/access.log' | goaccess --no-global-config --log-format=VCOMBINED --real-time-html -o /home/user/2.html --port=9870 --fifo-in=/tmp/2.in --fifo-out=/tmp/2.out --daemonize

Both reports updated with the data relevant to each report (data wasn't mixed). Please note that --daemonize won't make goaccess run in the background due to the unclosed pipe. Also, note that it's important not to use the single dash - and use absolute paths.

Let me know how it goes for you. Tests were ran on an arch linux box (4.9.45-1-lts x86_64)