Add Goaccess

[riazarbi]

Goaccess gives us a good way to see how much a WUI is being used. As the number of static sites increases, this will become more and more important - it'll give us a way of finding out whether a tool should be retired or not. It'll help us kill zombie reports.

[Gordonei]

A relevant consideration is that it looks like Docker doesn't keep container logs around for too long. This shouldn't matter, except that this container logs out the nginx access and error logs to STDOUT by default, which in turn gets persisted by the Docker daemon (and periodically rotated).

We have a couple of options:

1. Change the logging setup of docker_wui, and log to a volume which we run goaccess against.
2. Change the logging setup of Docker daemon.

I quite like (2), as it makes our setup more resilient - we'll be able to get the logs for any container from a single place.

[Gordonei]

According to this in the Docker docs, we could log to various options. I quite like the idea of something like Logstack, which is part of the ELK stack.

It means we could then using something like Kibana to visualise our logs :sunglasses:

[riazarbi]

I’m quite keen on sending these logs to a bucket so that an analyst can use whatever tools they want on them.

On Fri, 28 Jun 2019 at 17:49, Gordon Inggs notifications@github.com wrote:

According to this https://docs.docker.com/config/containers/logging/configure/ in the Docker docs, we could log to various options. I quite like the idea of something like Logstack https://www.elastic.co/products/logstash, which is part of the ELK stack.

It means we could then using something like Kibana to visualise our logs 😎

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cityofcapetown/docker_wui/issues/1?email_source=notifications&email_token=ADGPYN2333NU3FI6T3KKGMDP4YXH5A5CNFSM4HALT5CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY2OJZA#issuecomment-506782948, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGPYN2SGQB2YEDIDGJ3FQ3P4YXH5ANCNFSM4HALT5CA .

[Gordonei]

So that's the nice thing about using something like Logstash, as you can configure multiple output destinations - so you basically point the Docker logger at it, and then it'll forward the logs onto wherever you configure it.

[riazarbi]

A few questions -

Would the setup in this case be: Arbitrary container -> logs to docker -> logs to logstash -> elastisearch for analytics and minio for archiving?

If so, is the natural place to set up logstash on the VM directly?

Would we use one logstash daemon for all containers on a VM?

Are there any benefits to setting up logstash as a container that mounts docker logs read only?

On Sun, 30 Jun 2019 at 21:25, Gordon Inggs notifications@github.com wrote:

So that's the nice thing about using something like Logstash, as you can configure multiple output destinations - so you basically point the Docker logger at it, and then it'll forward the logs onto wherever you configure it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/cityofcapetown/docker_wui/issues/1?email_source=notifications&email_token=ADGPYN7TJDUQQGAABCYYHHLP5ECCDA5CNFSM4HALT5CKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY4SDZA#issuecomment-507060708, or mute the thread https://github.com/notifications/unsubscribe-auth/ADGPYN6K4OOINHBQ2Y55UZ3P5ECCDANCNFSM4HALT5CA .