search

microsoft / OMS-Agent-for-Linux

http://www.microsoft.com/oms
Other
410 stars 311 forks source link

Better Kubernetes support #235

Closed edevil closed 7 years ago

edevil commented 7 years ago

I'm trying to integrate the OMS agent in my Kubernetes cluster but there are certain things that could be improved. I'm using CoreOS, so I had to run the OMS agent in a privileged container as described in https://hub.docker.com/r/microsoft/oms/. The fact that I have to configure the Docker daemon to use the fluentd log driver instead of the json driver prevents certain commands like "docker logs" or "kubectl logs" from functioning properly.

Can't the agent tail the json log files present in /var/log/containers/*.log ?

Also, in the case of fluentd, there is a plugin that queries the Kubernetes API to inject additional metadata in the log lines. Does the OMS agent somehow make use of fluentd underneath? Can we use this plugin?

edevil commented 7 years ago

I managed to find the fluentd installation inside the OMS Agent, however I could not install the plugin due to an old version of Ruby:

ERROR:  Error installing fluent-plugin-kubernetes_metadata_filter:
    activesupport requires Ruby version >= 2.2.2.

Can I upgrade the Ruby version?

agup006 commented 7 years ago

Hi @edevil, We are looking to upgrade the version [2.2.2+] of Ruby in the OMS Agent for Linux in the next couple of months.

Today you can change the git module in the OMS Agent for Linux source to reference to Ruby v2.2.2+.

Thanks, Anurag

edevil commented 7 years ago

Are there any build instructions for the OMS agent?

jeffaco commented 7 years ago

@edevil, See the superproject, it has detailed build instructions for the OMS agent.

edevil commented 7 years ago

@jeffaco The instructions mention that if I don't have access to the "OMS TestConfig" repo I should only run the first git clone. However, even there I get denied access to another repository:

$git clone --recursive git@github.com:Microsoft/Build-OMS-Agent-for-Linux.git bld-omsagent
Cloning into 'bld-omsagent'...
...
Checking connectivity... done.
Submodule 'auoms' (git@github.com:Microsoft/OMS-Auditd-Plugin.git) registered for path 'auoms'
Submodule 'dsc' (git@github.com:Microsoft/PowerShell-DSC-for-Linux.git) registered for path 'dsc'
Submodule 'omi' (git@github.com:Microsoft/omi.git) registered for path 'omi'
Submodule 'omsagent' (git@github.com:Microsoft/OMS-Agent-for-Linux.git) registered for path 'omsagent'
Submodule 'opsmgr' (git@github.com:Microsoft/SCXcore.git) registered for path 'opsmgr'
Submodule 'opsmgr-kits' (git@github.com:Microsoft/SCXcore-osskits.git) registered for path 'opsmgr-kits'
Submodule 'pal' (git@github.com:Microsoft/pal.git) registered for path 'pal'
Cloning into '/Users/andre/work/vc/bld-omsagent/auoms'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:Microsoft/OMS-Auditd-Plugin.git' into submodule path '/Users/andre/work/vc/bld-omsagent/auoms' failed
jeffaco commented 7 years ago

@edevil Sorry, this change was made recently, and I was unaware of it. I'm actually not on the OMS project anymore. Thus, I'm not sure what OMS-Auditd-Plugin is, and why it's private.

@robbiezhang Can you give guidance here please?

keikhara commented 7 years ago

Hi @edevil,

this method has changed. You no longer need to set the --log-driver anymore. Please check our latest version. You will see the instruction has changed.

edevil commented 7 years ago

@keikhara Ok, that's good to know. What about the upgraded ruby version? I would like to be able to use the fluent-plugin-kubernetes_metadata_filter plugin in order to augment the log lines with Kubernetes specific information. However, it requires ruby >= 2.2.2.

keikhara commented 7 years ago

Hi @edevil,

for the ruby version, my colleague, @robbiezhang can provide more guidance. We are looking to support Kubernetes in future in addition to the Mesosphere DC/OS we have just supported.

regards,

Keiko

OguzPastirmaci commented 7 years ago

@edevil, for monitoring the containers in the cluster, you can deploy the OMS agent running as a container to Kubernetes using this DaemonSet:

https://gist.github.com/OguzPastirmaci/bba85a47c7c5fb71dc6e6f2acdd13249

edevil commented 7 years ago

@OguzPastirmaci Yes, that is what I'm planning to do. But I still need to be able to use the fluentd plugin that retrieves additional data from the Kubernetes API.

keikhara commented 7 years ago

@edevil @OguzPastirmaci

I will consider this an enhancement which we will be working on. regards,

Keiko

edevil commented 7 years ago

I've noticed that there's a commit 210ccc2 from October 2016 that upgrades the Ruby version of the OMS agent, however I've just tried the ruby version on the latest Microsoft/OMS docker image and it's still 2.2.0:

root@631c05c91553:/opt# ./microsoft/omsagent/ruby/bin/ruby -v
ruby 2.2.0p0 (2014-12-25) [x86_64-linux]

When will this commit reach a release?

lagalbra commented 7 years ago

@edevil We're looking to release a version this month that contains the update to Ruby 2.3.3, originating from commit https://github.com/Microsoft/OMS-Agent-for-Linux/commit/210ccc2247b463439ddb196a60e3acc095e8052b

lagalbra commented 7 years ago

@keikhara can speak more to better Kubernetes support.

As for using Ruby 2.3.3, the latest agent release does include Ruby 2.3.3.

keikhara commented 7 years ago

@edevil kubernetes support is there as of 2weeks ago. We are capturing k8s events.

edevil commented 7 years ago

@keikhara Which tag should I be using of these? https://hub.docker.com/r/microsoft/oms/tags/

Is there any additional configuration that I have to provide to the agent?

Thanks!

edevil commented 6 years ago

ping @keikhara

is there some documentation about what the agent is actually capturing? I don't see this additional information on the OMS portal. On another note, how does the OMS agent know it is running on a Kubernetes cluster? Does it assume from the computer name? https://github.com/Microsoft/Docker-Provider/blob/c79a123b4f579176bc7ff000a2a69752a49e4679/source/code/providers/Container_HostInventory_Class_Provider.cpp#L145 I have tried this but still no additional info on OMS.

keikhara commented 6 years ago

+Sambit and Joby

From: André Cruz [mailto:notifications@github.com] Sent: Friday, October 27, 2017 8:58 AM To: Microsoft/OMS-Agent-for-Linux OMS-Agent-for-Linux@noreply.github.com Cc: Keiko Harada keikhara@microsoft.com; Mention mention@noreply.github.com Subject: Re: [Microsoft/OMS-Agent-for-Linux] Better Kubernetes support (#235)

ping @keikharahttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkeikhara&data=02%7C01%7Ckeikhara%40microsoft.com%7C56f07f39037e45ef5ee408d51d53be33%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636447167869674856&sdata=ZL1x%2FImcwHMDQc5NTnB4nEd0lMm%2Fh2HxW6AL2VyoHmg%3D&reserved=0

is there some documentation about what the agent is actually capturing? I don't see this additional information on the OMS portal. On another note, how does the OMS agent know it is running on a Kubernetes cluster? Does it assume from the computer name? https://github.com/Microsoft/Docker-Provider/blob/c79a123b4f579176bc7ff000a2a69752a49e4679/source/code/providers/Container_HostInventory_Class_Provider.cpp#L145https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FDocker-Provider%2Fblob%2Fc79a123b4f579176bc7ff000a2a69752a49e4679%2Fsource%2Fcode%2Fproviders%2FContainer_HostInventory_Class_Provider.cpp%23L145&data=02%7C01%7Ckeikhara%40microsoft.com%7C56f07f39037e45ef5ee408d51d53be33%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636447167869674856&sdata=8Q1ShtJYqyXvAdp%2FddQdgOJ8CRztDo8NsXckkkQYM1w%3D&reserved=0 I have tried this but still no additional info on OMS.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2FOMS-Agent-for-Linux%2Fissues%2F235%23issuecomment-340012109&data=02%7C01%7Ckeikhara%40microsoft.com%7C56f07f39037e45ef5ee408d51d53be33%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636447167869674856&sdata=j2Gkt1gHifPBnm1knsfKdXmU%2Fp2Ag0COvHpe%2FOXM1Ck%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FASk8l1gdGNvBjm7lllrQyDyYnocJAmdRks5swf1rgaJpZM4KK9eW&data=02%7C01%7Ckeikhara%40microsoft.com%7C56f07f39037e45ef5ee408d51d53be33%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636447167869674856&sdata=IGD3U3uGdcJAk1J5URAcsQhCK3uUEoRLclni0Ywj65Y%3D&reserved=0.

edevil commented 6 years ago

Anyone?

jpoon commented 6 years ago

I haven't tried this myself, but I noticed there's a helm chart to install the OMS daemonset (https://kubeapps.com/charts/stable/msoms). @edevil, did you already try this? You can see from the volume mounts that it's grabbing data from the docker socket and /var/log.

edevil commented 6 years ago

I haven’t tried the helm chart, but the daemonset already mounts those dirs.

cricketfan5 commented 6 years ago

@here Can someone help me with the below issue ? I am installing the omsagent as a daemonset kubernetes hosts. I was able to install successfully with no error at the time of installation but it is not turning in green(green tick mark in the portal stating it is being monitored). And I see below error in the this path

Error:Unsupported Operation system: COREOS 1632.2.1. I am installing the OMSagent version(1.6.0-42) on COREOS version VERSION=1576.5.0 VERSION_ID=1576.5.0 BUILD_ID=2018-01-05-1121 PRETTY_NAME="Container Linux by CoreOS 1576.5.0 (Ladybug)" Any help is appreciated. Thanks in advance