Logging Infrastructure for K8s Clusters

[artntek]

Goals - Install a k8s log framework that...

• aggregates all logs (application, system, kubernetes control plane)
• streams a copy to our existing central syslog server for archival purposes (e.g. for forensic analysis on access logs, etc.)
• Streams a copy to another non-archival store (e.g. Elasticsearch), and provides tools for debugging and statistical analysis (e.g. Kibana)

[artntek]

I did a bunch of research and prototyping, and discovered that, in a modern k8s environment, the ELK stack (Elasticsearch - Logstash - Kibana) has been superseded by the EFK stack, where Logstash is replaced by Fluentd, FluentBit or FileBeat.

• Elasticsearch is a free, open-source search and analytics engine (similar in concept to solr), based on the Apache Lucene library.
• Kibana is a data visualization and exploration tool, used for log and time-series analytics, application monitoring, and operational intelligence.
• Fluentd, FluentBit & FileBeat are all logging agents (see below)

Simple overview:

1. Our k8s applications log to stdout, and this output is collected on the host Node machine, under /var/log/contianers/*.log (standard k8s functionality).
2. We need to install logging agents that do 3 things:
   1. aggregate logs from each of the node machines
   2. parse and transform them into the required structured logging format
   3. send copies to elasticsearch and to the central syslog archive.
3. Use the tools provided by Kibana to view and analyze the logs that are in elasticsearch.

Steps to deploy

1. install the Elasticsearch Bitnami chart, with the Kibana subchart enabled
2. install the Fluent Bit Bitnami chart
3. Note there is some config that needs to be changed in each case

[artntek]

Choice of Logging Agents:

Fluent Bit

...is essentially a faster, smaller-footprint replacement for Fluentd, and is much better suited to k8s.

FileBeat

I tried setting this up, but was unsuccessful in getting the pods to run without restarting - so I switched to Fluent Bit. mostly because there was a bitnami chart available. I did notice, after the fact, that there were some log entires in ES that I think must have come from FileBeat, despite it's instability, so who knows.

rsyslog

This also seems to be an option (especially if it is already running on our Node hosts), but I have not tried it yet.

syslog-ng

Supposed to be a super-duper version of rssylog. Didn't try it

[artntek]

see PR #45

[artntek]

Suggested next steps

Fluent Bit

• Inputs: Read and do this next: Fluent Bit Kubernetes
• Parsers & Filters: to structure logs into fields, and handle java multiline/stacktraces
  • Note there are multiple log formats to deal with - within apps (e.g. metacat log4j versus access log), across apps (e.g. is metacat log4j format same as metadig's?), and between app logs and system-level logs (k8s and linux)
• Output: split the stream and send a copy to Nick’s syslog server, in addition to ES
  1. https://docs.fluentbit.io/manual/pipeline/outputs/syslog
  2. Nick said: "anything on our network can push logs to 128.111.85.44:514"

FileBeat:

• Since we've used FileBeat before, is it worth pursuing FileBeat instead of adopting another solution (Fluent Bit)?