Create and put into place a way to monitor & alert on the NERC Switches

[joachimweyl]

Motivation

When one of the switches went down in NERC it turned out it had been down for multiple days. We realized we needed a more robust way to monitor switches.

Completion Criteria

Monitoring and alerting tested and working for the switches on the NERC network.

Description

• [x] decided on how to monitor the switches
• [ ] design the automated feature for monitoring and alerting
• [ ] Test
• [ ] put into place the monitoring and alerting tool

Completion dates

Desired - 2024-01-25 Required - TBD

[joachimweyl]

@jtriley is this something that Nick and/or Christian have discussed with you?

[hpdempsey]

This telemetry is also important for our researchers. They don't need access to the switches, but we do need to have access to at least the basic the telemetry for research (e.g. pps on VLANs etc.).

[joachimweyl]

We can only directly track the switches that are 100% NERC used, for switches shared with other projects we do not get access to them. @jtriley for the Switches that are 100% NERC do we have an automated alerting setup?

[jtriley]

No that will need to be implemented. The access is open though from the infra admin vlan via ICMP ping tests to the devices.

[schwesig]

@jtriley As discussed in the meeting right now: If pinging the switches is the only option, please add a list of pingable switches etc. and possible other ideas on how to "reach/monitor" them Thanks

[schwesig]

ideas:

• https://github.com/czerwonk/ping_exporter
• https://grafana.com/docs/grafana-cloud/monitor-public-endpoints/create-checks/checks/ping/
• https://github.com/prometheus/snmp_exporter
• https://www.youtube.com/watch?v=LDFalgu3Vtg

[schwesig]

https://docs.openshift.com/container-platform/4.13/observability/network_observability/network-observability-operator-release-notes.html

[schwesig]

• snmp
• OpenShift includes: installed network checker
• prometheus: targets

[jtriley]

@schwesig Apologies for the delay I had to get some ACLs in place but ICMP should now be open to the switches from the observability cluster hosts. These are the current IPs to monitor via ping test:

10.30.4.[10-14]

[schwesig]

still working on figuring out if a simple ping could do the job, or if we need snmp, or focussing on the flow, or on https://docs.openshift.com/container-platform/4.13/observability/network_observability/network-observability-operator-release-notes.html

so far I am focussing on the ping, as entry solution.