Add netstat error statistics

[njam]

netstat --statistics

E.g. SndbufErrors and RcvbufErrors. They only show up if present.

[…]
Ip:
    49942134 total packets received
    890 with invalid addresses
    0 forwarded
    0 incoming packets discarded <----------
    49889719 incoming packets delivered
    49813641 requests sent out
    71 outgoing packets dropped <----------
[…]
Udp:
    5964 packets received
    93831 packets to unknown port received.
    0 packet receive errors
    5669588 packets sent
    SndbufErrors: 12345 <----------
    RcvbufErrors: 12345 <----------
[…]

(this output might be different for different "netstat" versions!)

Maybe also relevant:

• /proc/net/udp
• ifconfig

Some input: http://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/#monitoring-network-devices

[njam]

Some input: http://blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/#monitoring-network-devices

[ppp0]

While researching about this topic, I came across some other methods to gather network errors..

e.g.

# bulldog4 (em1 main NIC)
$ ls /sys/class/net/em1/statistics
collisions           rx_dropped           rx_missed_errors     tx_carrier_errors    tx_heartbeat_errors
multicast            rx_errors            rx_over_errors       tx_compressed        tx_packets
rx_bytes             rx_fifo_errors       rx_packets           tx_dropped           tx_window_errors
rx_compressed        rx_frame_errors      tx_aborted_errors    tx_errors            
rx_crc_errors        rx_length_errors     tx_bytes             tx_fifo_errors     

# janus5
$ ls /sys/class/net/eth4/statistics/
collisions     rx_crc_errors   rx_frame_errors   rx_packets     tx_compressed   tx_heartbeat_errors
multicast      rx_dropped      rx_length_errors  tx_aborted_errors  tx_dropped      tx_packets
rx_bytes       rx_errors       rx_missed_errors  tx_bytes       tx_errors       tx_window_errors
rx_compressed  rx_fifo_errors  rx_over_errors    tx_carrier_errors  tx_fifo_errors

These values depend on the network card driver being loaded and used. (Needs ethtool to find out)

E.g.

# on bulldog:
$ ethtool -i em1
driver: tg3
version: 3.137
firmware-version: 5719-v1.41 NCSI v1.3.7.0
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

# on janus5 (10Gbit)
$ ethtool -i eth4
driver: ixgbe
version: 3.19.1-k
firmware-version: 0x80000208
bus-info: 0000:81:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Am I right to think that we're interested in:

• rx_dropped
• tx_dropped
• rx_error
• tx_error

Also, there is this:

$ cat /proc/net/snmp | grep Udp: 
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors InCsumErrors
Udp: 483476138 415872 201069 3355572970 201068 0 1

which would yield RcvbufErrors and SndbufErrors even when their value is 0

[ppp0]

let's wait for CI @njam please review

[njam]

• plugin/network.rb: Aren't these metrics counters (instead of gauges) - i.e. they increase over time?
• plugin/network.rb: "errors" sounds a bit harsh - how about "statistics"? Also let's avoid using abbreviations for variables ("e")

[njam]

As discussed let's iterate over all interfaces and sum up statistics.

[ppp0]

To be excluded:

• Local interface
• Bonded interfaces, as they are the same interface (in our case) e.g.

~$ cat /sys/class/net/{bond0,eth0}/statistics/tx_packets 
473150133
473150133

• virtual interfaces such as vboxnet0

May be make the interfaces to be excluded configurable per regex?

[ppp0]

@njam please review

[njam]

• plugin/network.rb: Now it's probably clearer and more flexible to define all metrics explicitly?
  • metrics_schema: "type" has a typo, let's add a unit
• monitor: Add a function that returns the sum of a certain check for a list of interfaces? Use "reduce()"?

[ppp0]

Agree it looks much better now, @njam please review

[njam]

• metrics_schema: No need to define a variable metrics
• monitor: No need for data
• _statistics_sum: I meant to add a docu block for documenting arguments and return value. See this example.

[ppp0]

@njam true, now?

[njam]

• _statistics_sum: Add return docu
• metrics_schema: Maybe add units - "Errors", and the other one is "Packets"?

[njam]

• _interfaces: let's also add docu here

Generally we try to add doc-blocks to all functions, it helps with refactoring. In bipbip it's not done in many places yet.

[ppp0]

makes sense, @njam please review

[njam]

lgtm! please remove the dashboard before releasing