search

openwisp / netengine

Python abstraction layer for extracting information from network devices.
http://netengine.rtfd.org
Other
39 stars 17 forks source link

[change] NetJSON DeviceMonitoring compliance #2 #62

Open purhan opened 3 years ago

purhan commented 3 years ago

Closes #2, closes #64, closes #65 Resolves part of #60

purhan commented 3 years ago

@pandafy @nemesisdesign In the last commit (https://github.com/openwisp/netengine/pull/62/commits/d46b5685c4bf4f63a678687fb4cde8e499f6c9cd), I've made the to_dict and to_json compliant with NetJSON DeviceMonitoring. Please take a look.

Todo: We need to define new methods to collect as much monitoring info as possible.

pandafy commented 3 years ago

A lot of methods we were using are no longer required in DeviceMonitoring schema but I have not removed them ye

You can go ahead with this, since it will make it clear for us what methods are to be removed. We can always revert the commit if something went wrong.

Todo: We need to define new methods to collect as much monitoring info as possible.

instead of being exhaustive, let's start with minimal data. Check what data is being collected in openwrt-openwisp-monitoring and add support for it here. Otherwise we may spend time collecting every possible metric many of which might not be used in OpenWISP. We can always iterate to make it better.

purhan commented 3 years ago

Thanks @pandafy , I have updated the OpenWRT backend for now (will do AirOS later). Below you can see the comparison between netengine and openwrt-openwisp-monitoring. I've added a few more metrics but it may not be possible to add all. Now I need to fix some tests.

output from openwrt-openwisp-monitoring

```json { "type": "DeviceMonitoring", "general": { "local_time": 1623319682, "uptime": 178, "hostname": "OpenWrt" }, "dhcp_leases": [ { "mac": "04:0e:3c:ca:55:5f", "client_id": "01:04:0e:3c:ca:55:5f", "client_name": "purhan", "ip": "192.168.2.140", "expiry": 1623362752 } ], "interfaces": [ { "type": "vlan", "up": false, "name": "eth0.2" }, { "mac": "1c:3b:f3:10:0a:42", "type": "bridge", "stp": false, "up": true, "txqueuelen": 1000, "name": "br-lan", "multicast": true, "addresses": [ { "proto": "static", "address": "192.168.2.1", "family": "ipv4", "mask": 24, "gateway": "192.168.0.1" }, { "family": "ipv6", "mask": 64, "proto": "static", "address": "fe80::1e3b:f3ff:fe10:a42" } ], "bridge_members": [ "eth0.1" ], "mtu": 1500 }, { "mac": "1c:3b:f3:10:0a:42", "type": "wireless", "up": true, "txqueuelen": 1000, "name": "wlan0", "wireless": { "country": "00", "noise": 0, "ssid": "TP-Link", "channel": 2, "tx_power": 20, "mode": "station", "signal": -46, "frequency": 2417 }, "multicast": true, "addresses": [ { "proto": "dhcp", "address": "192.168.0.100", "family": "ipv4", "mask": 24, "gateway": "192.168.0.1" }, { "family": "ipv6", "mask": 64, "proto": "static", "address": "fe80::1e3b:f3ff:fe10:a42" } ], "mtu": 1500 }, { "mac": "1c:3b:f3:10:0a:42", "type": "other", "up": true, "txqueuelen": 1000, "name": "eth0", "multicast": true, "addresses": [ { "family": "ipv6", "mask": 64, "proto": "static", "address": "fe80::1e3b:f3ff:fe10:a42" } ], "mtu": 1500 }, { "mac": "1c:3b:f3:10:0a:42", "type": "vlan", "up": true, "txqueuelen": 1000, "name": "eth0.1", "multicast": true, "mtu": 1500 } ], "resources": { "memory": { "total": 61452288, "shared": 278528, "free": 33243136, "cached": 7983104, "available": 26767360, "buffered": 2465792 }, "cpus": 1, "disk": [ { "filesystem": "/dev/root", "available_bytes": 0, "mount_point": "/rom", "used_percent": 100, "size_bytes": 2621440, "used_bytes": 2621440 }, { "filesystem": "/dev/mtdblock5", "available_bytes": 192512, "mount_point": "/overlay", "used_percent": 95, "size_bytes": 3735552, "used_bytes": 3543040 } ], "load": [ 0.33, 0.27, 0.11 ], "swap": { "free": 0, "total": 0 } }, "dns_servers": [ "8.8.8.8", "8.8.4.4" ], "neighbors": [ { "mac": "04:0e:3c:ca:55:5f", "state": "REACHABLE", "interface": "br-lan", "ip": "192.168.2.140" }, { "mac": "84:d8:1b:62:a3:55", "state": "REACHABLE", "interface": "wlan0", "ip": "192.168.0.1" }, { "mac": "04:0e:3c:ca:55:5f", "state": "STALE", "interface": "br-lan", "ip": "fe80::2ed4:5d74:941f:ce1e" } ] } ```

output from netengine

```json { "type": "DeviceMonitoring", "general": { "uptime": 30, "local_time": 1623319570 }, "resources": { "cpus": 0, "memory": { "total": 60012, "shared": 88, "free": 27340, "cached": 7160 }, "swap": { "total": 0, "free": 0 } }, "interfaces": [ { "name": "lo", "statistics": { "mac": "", "type": "softwareLoopback", "up": true, "rx_bytes": 2437, "tx_bytes": 2437, "mtu": 65536 } }, { "name": "eth0", "statistics": { "mac": "1c:3b:c3:b3:10:0a", "type": "ethernetCsmacd", "up": true, "rx_bytes": 160673, "tx_bytes": 615795, "mtu": 1500 } }, { "name": "Device 14c3:7662", "statistics": { "mac": "", "type": "ethernetCsmacd", "up": false, "rx_bytes": 0, "tx_bytes": 0, "mtu": 1500 } }, { "name": "br-lan", "statistics": { "mac": "1c:3b:c3:b3:10:0a", "type": "ethernetCsmacd", "up": true, "rx_bytes": 147924, "tx_bytes": 601205, "mtu": 1500 } }, { "name": "eth0.1", "statistics": { "mac": "1c:3b:c3:b3:10:0a", "type": "ethernetCsmacd", "up": true, "rx_bytes": 147924, "tx_bytes": 614999, "mtu": 1500 } }, { "name": "wlan0", "statistics": { "mac": "1c:3b:c3:b3:10:0a", "type": "ethernetCsmacd", "up": true, "rx_bytes": 604709, "tx_bytes": 158937, "mtu": 1500 } } ], "neighbors": [ { "mac": "04:0e:3c:c3:8a:55", "state": "REACHABLE", "interface": "br-lan" }, { "mac": "04:0e:3c:c3:8a:55", "state": "REACHABLE", "interface": "br-lan" }, { "mac": "c2:84:c3:98:1b:62", "state": "STALE", "interface": "wlan0" } ] } ```

coveralls commented 3 years ago

Pull Request Test Coverage Report for Build 938832440

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

Changes Missing Coverage Covered Lines Changed/Added Lines %
netengine/backends/snmp/airos.py 75 78 96.15%
netengine/backends/snmp/openwrt.py 104 108 96.3%
<!-- Total: 212 219 96.8% -->
Files with Coverage Reduction New Missed Lines %
netengine/backends/snmp/airos.py 4 96.15%
netengine/backends/snmp/openwrt.py 8 94.22%
<!-- Total: 12 -->
Totals Coverage Status
Change from base Build 900223237: 6.6%
Covered Lines: 600
Relevant Lines: 626
💛 - Coveralls
purhan commented 3 years ago

Missing things we found out in AirOS during our last call:

* device name

* device model (Ubiquiti LiteBeam ...)

* operating system (Linux version (withou date) + firmware version)

* please check what the mac addresses are and if we can display them (if these mac addresses are coming from the interfaces, we should include them)

@nemesisdesign Should this information be returned in the to_json function? Or should we just collect it for now? I'm unsure of where to put it in the JSON.

pandafy commented 3 years ago

@nemesisdesign Should this information be returned in the to_json function?

Yes. You can add this with 'system_info' key in the returned value. Moreover, it won't take much time if we want to move it around in the returned data structure if needed.