Kubernetes pod log files with acs-engine

[shayansarkar]

When I deploy a kubernetes 1.6.2 cluster using acs-engine it does not seem to be generating a log file for each container. If I ssh into a node and go to /var/lib/docker/containers/ I can see a list of all of the containers running on the node. Inside each of the directories are the config files and other information for the container. However, the container's log file seems to be missing in the 1.6.2 cluster. In a cluster created using "az acs" which creates a kubernetes 1.5.3 cluster, this directory also contains the log file of the container. However, in the acs-engine created cluster, the log files aren't there.

The main difference I could see is in the config.v2.json file that is in each of the /var/lib/docker/containers/ directories, the LogPath is set to "" in the 1.6.2 cluster but in the 1.5.3 cluster, the full path is specified.

I had been using these log files with a fluentd DaemonSet to aggregate all of the pod logs, but since they are no longer there in 1.6.2, the logging aggregation fails to work.

[JackQuincy]

Thanks for reporting this. We will get to this when we can. In the mean time if you or anyone else wants to submit a PR that would be awesome.

@colemickens If this was by design can you update this thread?

[rjtsdl]

@shayansarkar , for the logging. We made a few changes to completely use journald to manage the logs.

The default docker log driver is json-file, which would be the one you see under /var/lib/docker/containers/. We changed that to journald, to better manage logs. You can use docker logs, or kubectl logs to get the log.

For your reference, check out https://docs.docker.com/engine/admin/logging/overview/#supported-logging-drivers.

For the scenario, you want to push log to somewhere else. You may want to check out how journald push the log.

AFAIK, this is by design.

[seanknox]

@shayansarkar I'm going to close this thread as this is by design, but feel free to re-open if needed. More info about logging in Kubernetes can be found here: https://kubernetes.io/docs/user-guide/kubectl/v1.6/#logs

[thoyt]

Thanks @shayansarkar for reporting, and it would be great to reopen this is as a bug. Being able to choose which docker logging driver is used is critical, because Kubernetes doesn't do log aggregation. With journald, there isn't a reasonable way to integrate with Kibana or Splunk, two popular log aggregation systems, which are crucial parts of a production Kubernetes cluster. This is because the journald driver stores the logs in a binary format, which can't be used as-is with a Splunk forwarder, for example.

If there must be a default log driver, maybe json-file would be the best choice, since it would be easier to process and forward the logs to another service. kubectl logs would still work just fine too. Ideally, it would be possible to use any of the log drivers supported by Docker.

[seanknox]

Shipping Kubernetes master and worker logs is definitely doable, but up to the cluster admins to configure. It would be great to build this into ACS natively at some point. Here's more info: https://deis.com/blog/2016/off-cluster-kubernetes-logging-sumo-logic-and-logentries/

[shayansarkar]

Looking through some of the logging options in Docker as well as what makes sense from a cluster perspective, it would be nice to be able to specify the docker logging configuration when setting up a cluster with acs-engine. I agree that my original ticket isn't a bug, as journald is now the default option for docker. But since there are the other logging options, and some that may work better for a Kubernetes cluster, it would be great to have an easy way to expose those options in the template. Especially since a lot of people use fluentd and elasticsearch/kibana for logging aggregation. And the standard DaemonSet configuration from fluend's site uses the json-file logs.

[DonMartin76]

This also bites us. Our current configuration also relies on json-file and fluentd to forward to a Graylog instance... Not good, for us, right now.

[chartotu19]

I had the same issue with k8s v1.6.2 , docker daemon starts with --log-driver=journald . Currently, I am using this fluentd plugin to read journald logs via fluentd. https://github.com/reevoo/fluent-plugin-systemd

[shayansarkar]

@chartotu19 did you need to build your own fluentd docker image with that plugin from the docker hub image? https://hub.docker.com/r/fluent/fluentd/

[JackQuincy]

Hello so we have switched back to using docker json-file. This should support any fluentd based solution including OMS put out by MSFT. https://github.com/Microsoft/OMS-docker/tree/master/Kubernetes

[chartotu19]

@shayansarkar Yes, I am running a different pod with fluentd ( systemd plugin baked into it in dockerfile) on it to extract the logs from the k8s node.

[chartotu19]

@JackQuincy any reason why acs-engine switched back to use json-file? @rjtsdl mentioned that journald was better managing container logs. It would be nice to know the reason behind the decision :)

[JackQuincy]

We switched for better integration with this party software. Like fluentd. We started to use the docker log rotation options and it appears to be in a good state currently.

[DonMartin76]

I like this party software, rock on! (scnr)