Fault Management (Analysis and Handling)

shyam77git commented 11 months ago

Basic Information (context) Any failure (or an error) impacting a system/chassis or a sub-system is regarded as a fault. Broadly classified into SW (Software) and HW (Hardware) faults:

SW faults are the ones that can occur during SW processing of a workflow at process/sub-system or a system level
HW faults are those that can occur during SW or HW processing of a workflow at HW (board) level - e.g. HW component/device etc. They may occur at any of the following stages of system's functioning:
system configuration, bring-up
feature enablement/configuration
during steady state
feature disablement/unconfiguration
while going-down (config reload, reboot etc.)

Present State In SONiC, Fault is represented via an Event or an Alarm. SONiC has Event Framework HLD which can help event-detector to publish its event to the eventD redisDB. However, there is no Fault Manager/Handler which can take the needed/ platform-specified action(s) to recover the system from the generated fault.

Need for this feature This feature aims at adding a generic FM (Fault Management) Infrastructure which can do the following:

Abstract the platform/HWSKU nuances from an open source NOS (i.e. SONiC) by publishing platform-specific 'Fault-Action Policy table'
Fetch these events (alarms/faults) from the eventD (based on published YANG/schema)
Analyze them (in a generic way) against the above-mentioned Policy Table
Take action based on the lookup/match in Policy Table Action could either be generic or platform specfic

Benefits Platform supplied 'Fault-Action Policy table' has a holistic/system-level view of the platform (chassis/board/HWSKU) and can gauge the right action required to recover from the fault. It can either go with the recommended action (provided by the fault source/detector) or override it with the system-level one.

shyam77git commented 10 months ago

HLD (md) PR: https://github.com/sonic-net/SONiC/pull/1527

zhangyanzhao commented 8 months ago

Dell registered as the reviewer.

zhangyanzhao commented 8 months ago

Community review recording https://zoom.us/rec/share/G4YPod_DoyMGGc8RG-A6jakAEOwR4INXe8pfG5IXrDKZS5ozbyghJyXASgEthkZq.8EmVrFbqX2t3qan3

zhangyanzhao commented 8 months ago

@venkatmahalingam can you please let me know the github id of other reviewers from Dell? Thanks.

venkatmahalingam commented 8 months ago

@bhaveshdell @rathnasabapathyv @prvattem

sonic-net / SONiC

Fault Management (Analysis and Handling) #1520