Network usage for physical servers doesn't appear in grafana

[ckleban]

Hello,

When installing k8s, with heapster and inluxdb, network utilization appears in grafana for my VMs but not for my physical servers.

I'm using: v0.20.0-alpha12, heapster_influxdb:v0.5, heapster_grafana:v2.6.0-2.

My guess is that the network interfaces on the phsyical machines aren't being picked up correctly, but I'm not sure. My physical machines using a bond0 sub interface (bond0.3691, which the stats may or may not also appear in bond0 main interface) and my VMs use a eth0 interface.

My network interfaces on ubuntu VMs are (which show up in grafana): CA2KLE1V04K8N01 | SUCCESS | rc=0 >> docker0 Link encap:Ethernet HWaddr 02:42:e8:5e:ed:e1
inet addr:10.244.38.1 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::42:e8ff:fe5e:ede1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:57043 errors:0 dropped:0 overruns:0 frame:0 TX packets:60978 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11149744 (11.1 MB) TX bytes:95075935 (95.0 MB)

eth0 Link encap:Ethernet HWaddr 00:0c:29:0a:59:b6
inet addr:10.56.184.127 Bcast:10.56.184.255 Mask:255.255.255.0 inet6 addr: fe80::20c:29ff:fe0a:59b6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:443263 errors:0 dropped:48 overruns:0 frame:0 TX packets:124476 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:588347772 (588.3 MB) TX bytes:20500781 (20.5 MB)

flannel.1 Link encap:Ethernet HWaddr 0a:ea:96:4f:a9:05
inet addr:10.244.38.0 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::8ea:96ff:fe4f:a905/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:48009 errors:0 dropped:0 overruns:0 frame:0 TX packets:47230 errors:0 dropped:8 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2622062 (2.6 MB) TX bytes:11866415 (11.8 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

veth7538099 Link encap:Ethernet HWaddr 62:60:47:2d:55:c7
inet6 addr: fe80::6060:47ff:fe2d:55c7/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:54 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:4468 (4.4 KB)

veth32c0372 Link encap:Ethernet HWaddr ba:4b:2c:a8:9d:82
inet6 addr: fe80::b84b:2cff:fea8:9d82/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:38 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:3112 (3.1 KB)

veth4b5358c Link encap:Ethernet HWaddr 2e:04:9d:ef:c8:6e
inet6 addr: fe80::2c04:9dff:feef:c86e/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:42 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:3448 (3.4 KB)

veth833397e Link encap:Ethernet HWaddr 56:11:f0:fe:ed:d6
inet6 addr: fe80::5411:f0ff:fefe:edd6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:69 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:5774 (5.7 KB)

veth99b1b48 Link encap:Ethernet HWaddr c2:6c:f2:d0:cb:14
inet6 addr: fe80::c06c:f2ff:fed0:cb14/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:7234 errors:0 dropped:0 overruns:0 frame:0 TX packets:10647 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8496046 (8.4 MB) TX bytes:90437036 (90.4 MB)

vethc9ec562 Link encap:Ethernet HWaddr 8a:f4:2e:42:09:43
inet6 addr: fe80::88f4:2eff:fe42:943/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:45548 errors:0 dropped:0 overruns:0 frame:0 TX packets:45421 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3081348 (3.0 MB) TX bytes:3102558 (3.1 MB)

vethebdbecc Link encap:Ethernet HWaddr ba:7a:b2:23:ea:13
inet6 addr: fe80::b87a:b2ff:fe23:ea13/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:93 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:7830 (7.8 KB)

vethecf29c7 Link encap:Ethernet HWaddr 8e:70:c1:10:ed:42
inet6 addr: fe80::8c70:c1ff:fe10:ed42/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:4174 errors:0 dropped:0 overruns:0 frame:0 TX packets:5245 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:364595 (364.5 KB) TX bytes:1565476 (1.5 MB)

My network interfaces on physical servers running ubuntu (that do not appear) are: VA1KLE1V04K8N01 | SUCCESS | rc=0 >> bond0 Link encap:Ethernet HWaddr 5c:b9:01:89:7f:18
inet6 addr: fe80::5eb9:1ff:fe89:7f18/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:548734 errors:0 dropped:0 overruns:0 frame:0 TX packets:153974 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:745840709 (745.8 MB) TX bytes:81823807 (81.8 MB)

bond0.3691 Link encap:Ethernet HWaddr 5c:b9:01:89:7f:18
inet addr:10.136.191.14 Bcast:10.136.191.255 Mask:255.255.255.0 inet6 addr: fe80::5eb9:1ff:fe89:7f18/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:138271 errors:0 dropped:0 overruns:0 frame:0 TX packets:109654 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:718142603 (718.1 MB) TX bytes:78895997 (78.8 MB)

docker0 Link encap:Ethernet HWaddr 02:42:57:91:fe:1f
inet addr:10.244.41.1 Bcast:0.0.0.0 Mask:255.255.255.0 inet6 addr: fe80::42:57ff:fe91:fe1f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:57688 errors:0 dropped:0 overruns:0 frame:0 TX packets:68517 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6178145 (6.1 MB) TX bytes:33237405 (33.2 MB)

eth0 Link encap:Ethernet HWaddr 5c:b9:01:89:7f:18
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:548734 errors:0 dropped:0 overruns:0 frame:0 TX packets:153974 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:745840709 (745.8 MB) TX bytes:81823807 (81.8 MB)

eth1 Link encap:Ethernet HWaddr 5c:b9:01:89:7f:19
BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

flannel.1 Link encap:Ethernet HWaddr da:18:70:7a:3b:5c
inet addr:10.244.41.0 Bcast:0.0.0.0 Mask:255.255.0.0 inet6 addr: fe80::d818:70ff:fe7a:3b5c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:62698 errors:0 dropped:0 overruns:0 frame:0 TX packets:52517 errors:0 dropped:8 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:30007304 (30.0 MB) TX bytes:6896825 (6.8 MB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:2 errors:0 dropped:0 overruns:0 frame:0 TX packets:2 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:176 (176.0 B) TX bytes:176 (176.0 B)

veth0245420 Link encap:Ethernet HWaddr 4e:f3:99:57:c4:41
inet6 addr: fe80::4cf3:99ff:fe57:c441/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:221 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:18706 (18.7 KB)

veth6431319 Link encap:Ethernet HWaddr b6:a2:54:e4:98:5d
inet6 addr: fe80::b4a2:54ff:fee4:985d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:330 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:28056 (28.0 KB)

veth6828898 Link encap:Ethernet HWaddr 1e:e3:2d:c5:0b:12
inet6 addr: fe80::1ce3:2dff:fec5:b12/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:98 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:8200 (8.2 KB)

veth9697223 Link encap:Ethernet HWaddr 7e:74:a7:be:6d:ed
inet6 addr: fe80::7c74:a7ff:febe:6ded/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:8 errors:0 dropped:0 overruns:0 frame:0 TX packets:48 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:648 (648.0 B) TX bytes:3952 (3.9 KB)

veth9820230 Link encap:Ethernet HWaddr 5a:fe:87:79:6d:02
inet6 addr: fe80::58fe:87ff:fe79:6d02/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:283 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:24018 (24.0 KB)

veth023b0fe Link encap:Ethernet HWaddr 9e:47:72:d7:b1:02
inet6 addr: fe80::9c47:72ff:fed7:b102/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1 RX packets:3 errors:0 dropped:0 overruns:0 frame:0 TX packets:203 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:258 (258.0 B) TX bytes:17166 (17.1 KB)

[CBR09]

Hi, any workaround for that?. It will useful to me.

[DirectXMan12]

This sounds like a cAdvisor question, not a Heapster question -- does the node summary API report stats for your physical machines? If not, you should file this against Kubernetes and/or cAdvisor.

[CBR09]

Yes, the cAdvisor has reported stats for my physical machines. I got issue when my physical machine has multiple network interfaces, cAdvisor has reported network metrics of all network interfaces, but Heapster didn't, I don't know how to specify Heapster select right network interface.

[DirectXMan12]

Yes, the cAdvisor has reported stats for my physical machines.

But, does the node summary API report the right thing (curl $NODE:10250/stats/summary or $NODE:10255/stats/summary)? We just pull from the summary API more-or-less directly.

[CBR09]

curl http://x.x.x.x:10255/stats/ | jq '.stats[0].network'

{
  "tcp6": {
    "Closing": 0,
    "Listen": 0,
    "LastAck": 0,
    "Established": 0,
    "SynSent": 0,
    "SynRecv": 0,
    "FinWait1": 0,
    "FinWait2": 0,
    "TimeWait": 0,
    "Close": 0,
    "CloseWait": 0
  },
  "tcp": {
    "Closing": 0,
    "Listen": 0,
    "LastAck": 0,
    "Established": 0,
    "SynSent": 0,
    "SynRecv": 0,
    "FinWait1": 0,
    "FinWait2": 0,
    "TimeWait": 0,
    "Close": 0,
    "CloseWait": 0
  },
  "interfaces": [
    {
      "tx_dropped": 0,
      "name": "eno3",
      "rx_bytes": 0,
      "rx_packets": 0,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 0,
      "tx_packets": 0,
      "tx_errors": 0
    },
    {
      "tx_dropped": 0,
      "name": "eno4",
      "rx_bytes": 0,
      "rx_packets": 0,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 0,
      "tx_packets": 0,
      "tx_errors": 0
    },
    {
      "tx_dropped": 3427,
      "name": "flannel.1",
      "rx_bytes": 36771186124,
      "rx_packets": 59340479,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 248020988479,
      "tx_packets": 37138742,
      "tx_errors": 0
    },
    {
      "tx_dropped": 0,
      "name": "eno1",
      "rx_bytes": 0,
      "rx_packets": 0,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 0,
      "tx_packets": 0,
      "tx_errors": 0
    },
    {
      "tx_dropped": 0,
      "name": "enp3s0f4d1",
      "rx_bytes": 0,
      "rx_packets": 0,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 0,
      "tx_packets": 0,
      "tx_errors": 0
    },
    {
      "tx_dropped": 0,
      "name": "enp3s0f4",
      "rx_bytes": 685314787276,
      "rx_packets": 1243755757,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 706715945029,
      "tx_packets": 1490150638,
      "tx_errors": 0
    },
    {
      "tx_dropped": 0,
      "name": "eno2",
      "rx_bytes": 0,
      "rx_packets": 0,
      "rx_errors": 0,
      "rx_dropped": 0,
      "tx_bytes": 0,
      "tx_packets": 0,
      "tx_errors": 0
    }
  ],
  "tx_dropped": 0,
  "name": "eno3",
  "rx_bytes": 0,
  "rx_packets": 0,
  "rx_errors": 0,
  "rx_dropped": 0,
  "tx_bytes": 0,
  "tx_packets": 0,
  "tx_errors": 0
}

I can see cAdvisor has reported multiple interfaces in network.interfaces, but at the end of network I see one interface (eno3) that I think heapster picked it. How can I select it?, mean heapster select one of these in network.interfaces

[DirectXMan12]

ok, so I also just noticed this:

v0.20.0-alpha12

That's a really old version of Heapster. While I doubt this is a Heapster problem specifically, you should update so that we're working against an up-to-date Heapster. If you're not already using the summary API source, you should be using that, and then get the results from /stats/summary.

[CBR09]

I've used heapster v1.2.0, but I got issue as I mentioned above and I don't understand what you said about using the summary API source. My heapster configuration is: heapster --source=kubernetes:http://kube-api.kube-system:8080?inClusterConfig=false&useServiceAccount=false --sink=influxdb:http://influxdb.kube-system:8086?db=${INFLUX_DB}&user=${INFLUX_USER}&pw=${INFLUX_PASS}

[DirectXMan12]

so, to use the summary API, you need --source=kubernetes.summary_api instead. It's newer and better maintained, and avoids a whole class of problems.

[xiangpengzhao]

This issue may be due to cAdvisor. See my comment of the issue posted by @CBR09 in cAdvisor repo.

@ckleban @DirectXMan12

[CBR09]

@DirectXMan12 I've tried summary API, but I still got issue. @xiangpengzhao : thank you

[andyxning]

Once #1788 is merged. We this should not be a problem again.