Extend `vm-monitor` to monitor the tinyproxy

[sidoruka]

Background As we have to proxy egress traffic from the internal networks to the Internet - a tinyproxy is used. It may be overloaded with requests is some cases and start degrading. Administrators shall be aware of this and get a notification, similar to the node/filesystem/etc problems.

Approach

• Extend vm-monitor to monitor tinyproxy stats
• These statistics can be retrieved by requesting tinyproxy.stats URL with a cp-tinyproxy.default.svc.cluster.local proxy, e.g.:

  $ export http_proxy=http://cp-tinyproxy.default.svc.cluster.local
  $ curl -s tinyproxy.stats 
  {
  "server": "tinyproxy",
  "version": "1.10.0",
  "open": "1",
  "requests": "24",
  "bad": "0",
  "denied": "0",
  "refused": "0"
  }

• vm-monitor run such a query every N minutes, with the following configuration options:
  • Enable/Disable this monitoring feature (default: disabled)
  • URL (tinyproxy.stats)
  • HTTP proxy (http://cp-tinyproxy.default.svc.cluster.local)
  • Request schedule via a CRON string (* * * * *)
  • Thresholds for each of the stats JSON attributes (none of them are mandatory, so we can monitor only open or open and refused)
  • Resend delay (in minutes). As this parameter may be monitored each minute - we don't want to spam admins so often. Let's backoff from resending notifications for the same parameter for a defined period of time. This information shall be kept in-memory, no need to use a DB or other persistent storage.
• If any of the thresholds is reached - email shall be sent. The following information shall be included:
  • Event date/time
  • Parameter name (open)
  • Current parameter value (100)
  • Threshold value (50)

[sidoruka]

Please backport to release/0.16