Kubernetes Monitoring

[kjvalencik]

I am trying to use the docker agent to monitor containers running in Kubernetes. However, the agent assumes that when it is running in a container it will have a static name assigned to it. This is not possible if the agent is running inside the cluster. Instead, it will have a dynamic name based on a few bits of information.

Example:

k8s_agent.3a124fb0_scalyr-agent-kube1_kube-system_d1d11bd4b2f5fa9549577f46efe9fc20_408ef626

It would be great if the container_name parameter to the docker_monitor module also supported some sort of pattern matching. E.g.,

{
    "container_name": "*_scalyr-agent-*"
}

As a work around, I'm able to set "container_name" : "", but this causes the agent to monitor itself.

Another feature that would be great for monitoring K8S would be automatic parsing of container names in order to populate some meta data about the container. For example, the above name indicates that the name of the pod is scalyr-agent, it's running on the kube1 node in the kube-system namespace.

[steve-scalyr]

Thanks for getting in touch!

Would you have five minutes for a quick call to review your setup and confirm the technical details? Then we should be able to put together an updated agent for you.

Please let us know what would work for your schedule.

Cheers, Steve

On Fri, May 27, 2016 at 7:50 AM, K.J. Valencik notifications@github.com wrote:

I am trying to use the docker agent to monitor containers running in Kubernetes. However, the agent assumes that when it is running in a container it will have a static name assigned to it. This is not possible if the agent is running inside the cluster. Instead, it will have a dynamic name based on a few bits of information.

Example:

k8s_agent.3a124fb0_scalyr-agent-kube1_kube-system_d1d11bd4b2f5fa9549577f46efe9fc20_408ef626

It would be great if the container_name parameter to the docker_monitor module also supported some sort of pattern matching. E.g.,

{ "container_name": "_scalyr-agent-" }

As a work around, I'm able to set "container_name" : "", but this causes the agent to monitor itself.

Another feature that would be great for monitoring K8S would be automatic parsing of container names in order to populate some meta data about the container. For example, the above name indicates that the name of the pod is scalyr-agent, it's running on the kube1 node in the kube-system namespace.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/scalyr/scalyr-agent-2/issues/42, or mute the thread https://github.com/notifications/unsubscribe/ABDJerFWTBgp-u-DIb1NP3Z1yd_TExhjks5qFwTLgaJpZM4Iokya .

[seanwbren]

Hi @steve-scalyr , I'm also wondering if this * naming scheme is possible on the docker agent. Thanks!

[kjvalencik]

@sbberk I ended up solving this by setting an environment variable using the downward api.

http://kubernetes.io/docs/user-guide/downward-api/

[hjacobs]

@kjvalencik we are working on https://github.com/zalando-incubator/kubernetes-log-watcher to ship logs to Scalyr (requires Scalyr agent) --- maybe it helps (docs and more missing).

[jeffkimble]

+1 on out of the box K8S support.

[pierreozoux]

I did create this chart, if anybody is interested to try :) https://github.com/kubernetes/charts/pull/1066

[scottrigby]

@pierreozoux I drew from some of your earlier work for a new Scalyr Helm chart. Check out https://github.com/scalyr/scalyr-agent-2/issues/118#issuecomment-490135304 if you're still interested in this - always up for collaboration 😄