CORTX-28171 : System health state machine : Implementation

[mukhtar-inamdar]

Problem Statement

• System health state machine: Implementation
• i.e. Changes for new disk event types it can pass through event filtering and parsing.

Design

• Sent mock disk event with event_type "failed", "online", "repairing", "repaired", "rebalancing"
• and check if pass till system_health creation
• did require changes however, removed the changes required for resource type 'disk' as resource type changes are getting implemented as a part of another task
• tested the code changes see below details
• Sample system health event for the disk :
• {"source": "monitor", "event_id" : "164621220359222845229b4c8db9f5a4d08e838161", "event_type": "repairing", "severity": "informational", "site_id": "1", "rack_id": " 1", "cluster_id": "09b8b2df51be42d7a3c4185e9f871029", "storageset_id": "ssc-vm-g4-rhev4-0147.colo.seagate.com", "node_id": "ssc-vm-g 4-rhev4-0147.colo.seagate.com", "host_id": "ssc-vm-g4-rhev4-0147.colo.seagate.com", "resource_type": "disk", "timestamp": "164621220 3", "resource_id": "sdf", "specific_info": {"is_status": true}}
• Log snippet of fault tolerance:
• Note: In the below log, you can see the event is reached to health system update (no further changes will track in separate task-related health_system) so disk event for above-mentioned types are successfully filtered and parsed and creating health system event see above example

  2022-03-02 10:39:09 fault_tolerance [156]: INFO [init] All cluster element are loaded, Ready to process alerts ................. 2022-03-02 10:39:09 fault_tolerance [156]: INFO [init] ClusterResource Parser is initialized ... 2022-03-02 10:39:09 fault_tolerance [156]: INFO [filter_event] This alert needs an attention: resource_type: disk 2022-03-02 10:39:09 fault_tolerance [156]: ERROR [get_hierarchy] Health update hierarchy not present for component: disk

Coding

• [x] Coding conventions are followed and code is consistent

Testing

• [ ] Unit and System Tests are added
• [x] Test Cases cover Happy Path, Non-Happy Path, and Scalability
• [x] Testing was performed with RPM Note: check above testing details

Review Checklist

• [x] PR is self reviewed
• [x] JIRA number/GitHub Issue added to PR
• [x] Jira and state/status is updated and JIRA is updated with PR link
• [x] Check if the description is clear and explained
• [x] If yes for above point, is a notification sent to all other cortx components? [Y/N] N
• [x] Side effects on other features (deployment/upgrade)? [Y/N] N
• [x] Dependencies on other component(s)? [Y/N] N
• If yes for above point, post a link to the corresponding PR.

Review Checklist

• [ ] Is perfline test run and the report with and without the changes updated in the PR? [Y/N]:

Documentation

Checklist for Author

• [x] Changes done to WIKI / Confluence page / Quick Start Guide

[ArchanaLimaye]

Regarding point mentioned in design : "and check if pass till system_health creation" In case of state change, are these events getting published on the message bus to health_monitor ? Please confirm if not already done

2022-03-03 19:08:38 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/failed 2022-03-03 19:08:38 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/failed 2022-03-03 19:08:39 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/online 2022-03-03 19:08:39 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/online 2022-03-03 19:08:40 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/repairing 2022-03-03 19:08:40 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/repairing 2022-03-03 19:08:41 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/repaired 2022-03-03 19:08:41 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/repaired 2022-03-03 19:08:42 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/rebalancing 2022-03-03 19:08:42 health_monitor [7]: INFO [evaluate] Evaluated action [] for key action/disk/rebalancing