Use GoAccess for VM Log monitoring

[kevinquillen]

Issue Type

• Feature Idea

Summary

I would like to use GoAccess as a dev tool instead of PimpMyLog. How can I go about setting it up automatically when I create a new VM?

[geerlingguy]

@kevinquillen - The best way currently would be to add a task include file (to use Ansible tasks) or a shell script to run post-provision: http://docs.drupalvm.com/en/latest/extending/scripts/

If you end up finding GoAccess to be a good drop-in replacement for PML, then I'm definitely not opposed to the idea of switching. I'm not married to PML, as I never use it myself!

But does GoAccess display log entries in a centralized way, or is it just a stats dashboard?

[fubarhouse]

@geerlingguy GoAccess a dashboard of stats collected from input access logs.

Back when I was complaining about PML I switched over to GoAccess and found it incredibly valuable. It's got a more concise scope dealing strictly with access logs, and there is (minimal) configuration to happen - but it's a very useful tool.

I plugged 6 months worth of production logs into it 👍

[xbreid]

@geerlingguy Where do I sign the petition to get PML switched out with GoAccess? That would be awesome if you're still considering it!

[oxyc]

@fubarhouse do you have any example of your setup that you could post?

[fubarhouse]

@oxyc I'll add some information about it on Tuesday morning AEST.

[fubarhouse]

@oxyc,

Goaccess

Install

I usually build mine from source, however 1.2 is available via the system package management utility for all linux-based systems.

Building from source. Note that the folder created must not be moved or changed after building.

sudo apt-get -y install libncursesw5-dev gcc make
sudo apt-get -y install libgeoip-dev libtokyocabinet-dev
wget http://tar.goaccess.io/goaccess-1.2.tar.gz
tar -xzvf goaccess-1.2.tar.gz
cd goaccess-1.2
sudo ./configure --enable-utf8 --enable-geoip
sudo make
sudo make install
sudo ln -s /usr/local/bin/goaccess /usr/bin/goaccess

It can now be used from the console, but it also features some really cool HTML generation tools...

For the html work, you'd need to point a port to a location, or a virtual host to a location to serve it.

Config

The following three requirements must be specified in the goaccess.conf file.

The following format strings will need to be set and compatible with the input log files, here're the ones I'm using (compatible with drupal vm's access logs and acquia's access logs.

You will simply need to uncomment these values.

Time:

# The following time format works with any of the
# Apache/NGINX's log formats below.
#
time-format %H:%M:%S

Date:

# The following date format works with any of the
# Apache/NGINX's log formats below.
#
date-format %d/%b/%Y

Logs:

# NCSA Combined Log Format
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

Usage

Same of live information via console:

To get this, you'll need read access to any log files, and pipe them into goaccess:

tail -f /var/log/nginx/access.log | goaccess.

This must point to where the access logs are being collected, the image above points to cat /var/log/nginx/access.log.* | goaccess for me, however you can also do this for gzips - see the documentation for more info

The web interface:

I hope this helps :)

I've been able to identify some serious problems in production with the use of this tool, so it's permanently in my toolchain now.

[oxyc]

Thanks! Looks pretty easy to setup. And saw the the config allows multiple log-files so that's easy. Then just a simple systemd service to keep it running. Main issue I see is https://github.com/allinurl/goaccess/issues/213

[geerlingguy]

I like what I see. And PML has never had the uptake I thought it might have originally. So I'm considering swapping it out for Drupal VM's default—especially if I can get it to work out of the box (instead of having to do extra setup post-Drupal-VM-build!).

[fubarhouse]

@geerlingguy,

I've started working on a task file for the installation and configuration of GoAccess for you.

All I need to sort out now is the daemon, and test it via the supported vhosts variables.

I'll submit a PR once this is ready - I imagine you'll want some things about it changed.

For now, it's in /provisioning/tasks/goaccess.yml.

Edit:

A PR is ready for submission, however the following need attention:

• it appears when using apache the log files are empty, always...
• the daemon doesn't appear to be staying alive.

Preview:

[fubarhouse]

NGINX is pretty solid using the above log format, but I've noted that apache logs are found at /var/log/apache2/other_vhosts_access.log, using a similar but different log format.

I've made an adjustment to the config which is working now, but I'll need to test this against a RedHat httpd setup and change the daemon reference for when that is addressed.

I'll note that how I'm using this now is to have a cron regenerate the report every minute, but I don't believe that is the best solution for everybody - it's more for testing.

[geerlingguy]

It seems like PML's future is pretty uncertain, though I want to thank @potsky again for working on it, it was (and is, for PHP < 8) a great log inspection tool written in PHP.

But as I'm uncertain about the future of Drupal VM itself, and I'm deprecating my PML role, I'm planning on at least removing PML from Drupal VM entirely to lighten the maintenance burden and code footprint.

See: https://github.com/geerlingguy/drupal-vm/issues/2174

[frob]

Does GoAccess work with log files other than Access logs? What about Drupal syslog files or php error logs?