OMS Agent not sending data to Perf table in Log analytics

[ramon-tomas-c]

We are launching Jobs continuosly in Azure K8s Service every x minutes. When checking out the metrics (%CPU, memory, etc) in Perf table in Log analytics, does not appear the metrics for all the containers in Perf. It looks like OMS is not sending metrics for some containers and sometimes even not sending any metric at all during hours.

Version: -Azure AKS version 1.10.6 -OMS Linux Agent version 1.6.0-42 -Jobs based on .net Core 2.1

I have checked OMS agent logs and I haven't seen any failure.

**** Logs *** % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 2285 0 2285 0 0 26301 0 --:--:-- --:--:-- --:--:-- 26569

• Trying 10.0.0.1... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 10.0.0.1 (10.0.0.1) port 443 (#0)

• found 1 certificates in /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
• found 592 certificates in /etc/ssl/certs
• ALPN, offering http/1.1
• SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
• server certificate verification OK
• server certificate status verification SKIPPED
• common name: apiserver (matched)
• server certificate expiration date OK
• server certificate activation date OK
• certificate public key: RSA
• certificate version: #3
• subject: CN=apiserver
• start date: Tue, 07 Aug 2018 12:54:04 GMT
• expire date: Thu, 06 Aug 2020 12:54:04 GMT
• issuer: CN=ca
• compression: NULL
• ALPN, server accepted to use http/1.1

  GET /api/v1/nodes HTTP/1.1 Host: 10.0.0.1 User-Agent: curl/7.47.0 Accept: /

< HTTP/1.1 200 OK < Server: nginx/1.13.6 < Date: Tue, 07 Aug 2018 21:03:01 GMT < Content-Type: application/json < Transfer-Encoding: chunked < Connection: close < Strict-Transport-Security: max-age=15724800; includeSubDomains; < { [3876 bytes data]

100 22908 0 22908 0 0 54366 0 --:--:-- --:--:-- --:--:-- 54803

• Closing connection 0 v1.10.6 aks-agentpool-76627039-0 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 24520 0 24520 0 0 4775k 0 --:--:-- --:--:-- --:--:-- 5986k sha256:ac0ca11500013d1827b163cdda8c1f544ff87ca15f314e96e763101c07074900 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 3543 0 3543 0 0 342k 0 --:--:-- --:--:-- --:--:-- 384k [u'microsoft/oms:ciprod06072018']

/opt/omi/bin/omicli: result: MI_RESULT_FAILED

VERBOSE from OMS_MetaConfigHelper.py: Disable flag set to True, setting mof to disabled mode. VERBOSE from OMS_MetaConfigHelper.py: Successfully configured omsconfig. /opt/omi/bin/omiserver: server is not running

Workspace f5eef9f7-7249-4b00-bcd2-b4b91bf329d4 already onboarded and agent is running. Symbolic links have not been created; re-onboarding to create them info Generating certificate ... -e info Agent GUID is eea59e35-de95-4657-bf00-ca0ad3bfb2c0 -e info Onboarding success Configure syslog... Configuring rsyslog for OMS logging Restarting service: rsyslog invoke-rc.d: could not determine current runlevel

• Stopping enhanced syslogd rsyslogd ...done.
• Starting enhanced syslogd rsyslogd ...done. Configure heartbeat monitoring agent... Configure log rotate for workspace f5eef9f7-7249-4b00-bcd2-b4b91bf329d4... INFO: Configuring OMS agent service f5eef9f7-7249-4b00-bcd2-b4b91bf329d4 ... invoke-rc.d: could not determine current runlevel
• Starting Operations Management Suite agent (f5eef9f7-7249-4b00-bcd2-b4b91bf329d4): ...done. -e info Configured omsconfig instance of OMI_Identify { [Key] InstanceID=2FDB5542-5896-45D5-9BE9-DC04430AAABE SystemName=omsagent-k66cr ProductName=OMI ProductVendor=Microsoft ProductVersionMajor=1 ProductVersionMinor=4 ProductVersionRevision=2 ProductVersionString=1.4.2-2 Platform=LINUX_X86_64_GNU OperatingSystem=LINUX Architecture=X86_64 Compiler=GNU ConfigPrefix=GNU ConfigLibDir=/opt/omi/lib ConfigBinDir=/opt/omi/bin ConfigIncludeDir=/opt/omi/include ConfigDataDir=/opt/omi/share ConfigLocalStateDir=/var/opt/omi ConfigSysConfDir=/etc/opt/omi/conf ConfigProviderDir=/etc/opt/omi/conf ConfigLogFile=/var/opt/omi/log/omiserver.log ConfigPIDFile=/var/opt/omi/run/omiserver.pid ConfigRegisterDir=/etc/opt/omi/conf/omiregister ConfigSchemaDir=/opt/omi/share/omischema ConfigNameSpaces={root-scx, root-cimv2, root-oms, root-omi} } instance of Container_HostInventory { [Key] InstanceID=6ffaac9b-2dc6-4544-89cc-7b4bd47f1c9a Computer=aks-agentpool-76627039-0 DockerVersion=1.13.1 OperatingSystem=Ubuntu 16.04.5 LTS Volume=local Network=bridge host macvlan null overlay NodeRole=Agent OrchestratorType=Kubernetes }

• Starting periodic command scheduler cron ...done. Primary Workspace: f5eef9f7-7249-4b00-bcd2-b4b91bf329d4 Status: Onboarded(OMSAgent Running) omi 1.4.2.2 omsagent 1.6.0.42 docker-cimprov 2.0.0.3

  **** omsagent.log ****

2018-08-08 09:30:02 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.22s 2018-08-08 09:31:02 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:31:02Z 2018-08-08 09:31:03 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:31:03 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:31:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:31:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:31:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:31:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:31:05 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:31:22 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.6s 2018-08-08 09:32:23 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:32:23Z 2018-08-08 09:32:23 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:32:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:32:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:32:25 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:32:25 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:32:25 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:32:25 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:32:42 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.13s 2018-08-08 09:33:42 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:33:42Z 2018-08-08 09:33:42 +0000 [info]: in_container_sudo_tail : System Container list last refreshed at 2018-08-08T09:28:23+00:00 - refreshing now at 2018-08-08T09:33:42+00:00 2018-08-08 09:33:43 +0000 [info]: in_container_sudo_tail : System Container ID List: {"docker://b090a71c81f14c2e519b86cf6a771124f7e10d8eff90cd1b8376608488ba4518"=>"kube-system", "docker://548d74cb4f6c7f1466a14d0802ae0a40ae1e6ef9c2fa02274a8b4236b159f964"=>"kube-system", "docker://fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21"=>"kube-system", "docker://a2386252a422d32f1203ad289c4c7778ceb240d3dc62ed77acd4d3357c0c1b09"=>"kube-system", "docker://c3cd6e06a69b563e7e8e507bab17be016b33c8be1938d6228999bcaaa70ab74d"=>"kube-system", "docker://e3b59bd0a2436523de306a9826f6e84854d6cc96833a384ccabd5b8766556d21"=>"kube-system", "docker://af24950dadb803612bfbd1017f1ba62814d54070b96ee63f84ba4733758fdf25"=>"kube-system", "docker://64d056af040e4a71c43912c41bb92f4ff833435ad7229021d73da24658fb7038"=>"kube-system", "docker://a53480ae459ee968aea0a3d30d6b6779cea7d36df71cc5220889e6d6a5ac891c"=>"kube-system", "docker://b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548"=>"kube-system", "docker://68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002"=>"kube-system", "docker://2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b"=>"kube-system", "docker://84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143"=>"kube-system", "docker://ad8267930771525fb1628ad167bccd037a35c95c950f2863947b92e1bac3362e"=>"kube-system", "docker://000b9665db5f347e850357062aadd516e3db5fe48c468824d5e7003ba7114f42"=>"kube-system", "docker://89af0f0f081ada3ce7d68667ac6e7a2d014f5a565548a455f58198be039c6d5e"=>"kube-system", "docker://72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18"=>"kube-system", "docker://98276b171a85f730316f353ebb55d048412fe57d2b29356f8d6a6dddcdb69b35"=>"kube-system", "docker://2154ebf27d900710a4e674ec0283fb5ae70c6b5303e3975d4933c2cc6dce9e26"=>"kube-system", "docker://cfb266eb2b9dc7749bcb4b36bd8ee92c1d54122893eac57ffb63bf0bb2a79204"=>"kube-system", "docker://52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a"=>"kube-system", "docker://8502e6109f598323588df3e02d0916122ba14c08ee2cfd493e30916d6875b844"=>"kube-system", "docker://caa952398da5bb66a781ec602e24101be0cfb1f947b17508f4af35651e371336"=>"kube-system", "docker://142a701009da3c7ba1ba2889b623b63f3f8028499056250f61c546dc81868913"=>"kube-system", "docker://652ff855ae1c7c57e7a517fc26512a3765a4358421ee237a48a2ade23639687e"=>"kube-system"} 2018-08-08 09:33:43 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:33:43 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:33:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:33:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:33:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:33:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:33:45 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:34:02 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.21s 2018-08-08 09:35:01 +0000 [info]: Sending OMS Heartbeat succeeded at 2018-08-08T09:35:01.514Z 2018-08-08 09:35:02 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:35:02Z 2018-08-08 09:35:03 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:35:03 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:35:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:35:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:35:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:35:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:35:05 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:35:22 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.29s 2018-08-08 09:36:23 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:36:23Z 2018-08-08 09:36:23 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:36:23 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:36:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:36:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:36:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:36:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:36:25 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:36:42 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.13s 2018-08-08 09:37:42 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:37:42Z 2018-08-08 09:37:43 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:37:43 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:37:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:37:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:37:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:37:44 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:37:45 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:38:02 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.18s 2018-08-08 09:39:01 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:39:01Z 2018-08-08 09:39:01 +0000 [info]: in_container_sudo_tail : System Container list last refreshed at 2018-08-08T09:33:42+00:00 - refreshing now at 2018-08-08T09:39:01+00:00 2018-08-08 09:39:03 +0000 [info]: in_container_sudo_tail : System Container ID List: {"docker://b090a71c81f14c2e519b86cf6a771124f7e10d8eff90cd1b8376608488ba4518"=>"kube-system", "docker://548d74cb4f6c7f1466a14d0802ae0a40ae1e6ef9c2fa02274a8b4236b159f964"=>"kube-system", "docker://fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21"=>"kube-system", "docker://a2386252a422d32f1203ad289c4c7778ceb240d3dc62ed77acd4d3357c0c1b09"=>"kube-system", "docker://c3cd6e06a69b563e7e8e507bab17be016b33c8be1938d6228999bcaaa70ab74d"=>"kube-system", "docker://e3b59bd0a2436523de306a9826f6e84854d6cc96833a384ccabd5b8766556d21"=>"kube-system", "docker://af24950dadb803612bfbd1017f1ba62814d54070b96ee63f84ba4733758fdf25"=>"kube-system", "docker://64d056af040e4a71c43912c41bb92f4ff833435ad7229021d73da24658fb7038"=>"kube-system", "docker://a53480ae459ee968aea0a3d30d6b6779cea7d36df71cc5220889e6d6a5ac891c"=>"kube-system", "docker://b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548"=>"kube-system", "docker://68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002"=>"kube-system", "docker://2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b"=>"kube-system", "docker://84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143"=>"kube-system", "docker://ad8267930771525fb1628ad167bccd037a35c95c950f2863947b92e1bac3362e"=>"kube-system", "docker://000b9665db5f347e850357062aadd516e3db5fe48c468824d5e7003ba7114f42"=>"kube-system", "docker://89af0f0f081ada3ce7d68667ac6e7a2d014f5a565548a455f58198be039c6d5e"=>"kube-system", "docker://72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18"=>"kube-system", "docker://98276b171a85f730316f353ebb55d048412fe57d2b29356f8d6a6dddcdb69b35"=>"kube-system", "docker://32e450c6ac06145223006a123315f161b81587fb3a2fbe761712724beef23a34"=>"kube-system", "docker://cfb266eb2b9dc7749bcb4b36bd8ee92c1d54122893eac57ffb63bf0bb2a79204"=>"kube-system", "docker://52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a"=>"kube-system", "docker://8502e6109f598323588df3e02d0916122ba14c08ee2cfd493e30916d6875b844"=>"kube-system", "docker://caa952398da5bb66a781ec602e24101be0cfb1f947b17508f4af35651e371336"=>"kube-system", "docker://142a701009da3c7ba1ba2889b623b63f3f8028499056250f61c546dc81868913"=>"kube-system", "docker://652ff855ae1c7c57e7a517fc26512a3765a4358421ee237a48a2ade23639687e"=>"kube-system"} 2018-08-08 09:39:03 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:39:03 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:39:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:39:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:39:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:39:04 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:39:05 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:39:22 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.41s 2018-08-08 09:40:23 +0000 [info]: in_cadvisor_perf::run_periodic @ 2018-08-08T09:40:23Z 2018-08-08 09:40:23 +0000 [info]: Sending OMS Heartbeat succeeded at 2018-08-08T09:40:23.315Z 2018-08-08 09:40:23 +0000 [info]: in_container_sudo_tail : Excluding system container with ID b0da1341afa3774d4f83787848c79f33b64cd7b33d0aa55778bfbf80d493e548 from tailng for log collection 2018-08-08 09:40:23 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 72b056f75c6ff16f1e403953d6dfb94e2afa526d476e0c01db82ddc6aa07cb18 from tailng for log collection 2018-08-08 09:40:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 52bb1ef9fb71377035b9c45a1f81ad7fa9f9bc0395464739357ca1f65256314a from tailng for log collection 2018-08-08 09:40:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 2634e801a7b0a87d15a7a1ebbb22770f4e1f806ea5ec03361d97db7cb351881b from tailng for log collection 2018-08-08 09:40:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 84fcf4c2d313ac1a3b338e246de7f16ccdb4eb237f3b3b5a52dd6a80091e7143 from tailng for log collection 2018-08-08 09:40:24 +0000 [info]: in_container_sudo_tail : Excluding system container with ID fbbcc7f427d61b5189752c991543c339a667e95279a4202322773b2257415f21 from tailng for log collection 2018-08-08 09:40:25 +0000 [info]: in_container_sudo_tail : Excluding system container with ID 68c0452178c7520b35ec7c81ebc90d68ecb70553dd622454c27974351d916002 from tailng for log collection 2018-08-08 09:40:41 +0000 [debug]: Success sending LINUX_PERF_BLOB.LOGMANAGEMENT x 36 in 0.29s

[shpimpal]

@keikhara can you please take a look?