CORTX-28605 Subscribe to disk events and changes to ActionHandler to send disk alerts to Hare

[ArchanaLimaye]

Problem Statement

Handling needs to be added to Subscribe for disk events from Hare and to send disk alerts to Hare

Design

Changes added for Hare subscription for resource type disk in HA mini provisioning Action Handlers added for disk and cvg resource types

LLD at https://seagate-systems.atlassian.net/wiki/spaces/PUB/pages/891781277/Action+Status+Update+LLD

Coding

• [x] Coding conventions are followed and code is consistent

Testing

• [ ] Unit and System Tests are added
• [ ] Test Cases cover Happy Path, Non-Happy Path and Scalability
• [x] Testing was performed with RPM

https://jts.seagate.com/secure/attachment/506958/testing_details_CORTX-28605.txt

Review Checklist

• [x] PR is self reviewed
• [x] JIRA number/GitHub Issue added to PR
• [x] Jira and state/status is updated and JIRA is updated with PR link
• [x] Check if the description is clear and explained
• [ ] If yes for above point, is a notification sent to all other cortx components? [Y/N] NA
• [ ] Side effects on other features (deployment/upgrade)? [Y/N] N
• [ ] Dependencies on other component(s)? [Y/N] N
• If yes for above point, post link to the corresponding PR.

Review Checklist

• [ ] Is perfline test run and the report with and without the changes updated in the PR? [Y/N]:

Documentation

Checklist for Author

• [ ] Changes done to WIKI / Confluence page / Quick Start Guide

[ArchanaLimaye]

Testing details [Detailed logs added in https://jts.seagate.com/secure/attachment/506958/testing_details_CORTX-28605.txt]

1. Testing the event subscription :

   Deployed CORTX on 3N setup. Created HA RPM and installed it in the HA POD

   a. Checked the system health keys in consul using "consul kv get -http-addr=consul-server.default.svc.cluster.local:8500 --recurse cortx/ha/v1" b. Confirmed that the keys for disk resource are not present c. Executed ha _setup explicitly : /usr/lib/python3.6/site-packages/ha/k8s_setup/ha_setup.py config --services health_monitor --config yaml:///etc/cortx/cluster.conf d. Verified that the system health in consul shows all keys for disk subscription

cortx/ha/v1: cortx/ha/v1/action/disk/failed:["publish"] cortx/ha/v1/action/disk/online:["publish"] cortx/ha/v1/events/disk/failed:["hare"] cortx/ha/v1/events/disk/online:["hare"] cortx/ha/v1/events/subscribe/hare:["node/online", "node/failed", "disk/online", "disk/failed"] cortx/ha/v1/message_type/hare:ha_event_hare

2. Testing publish of disk event to Hare

   a. Using test code created disk online system health event and published it on message bus for message_type "health_events" b. Verified by looking at health_monitor.log file that this event is

   • Received by health monitor
   • processed and sent out to Hare c. Did the above steps for disk online and disk failed and verified that for both events , alert is sent to Hare d. Similarly created cvg online, cvg failed events and published it on message bus for message_type "health_events" e. verified that event is
   • Received by health monitor
   • Not sent out to Hare since subscribtion is not done for CVG resource type f. Explicitly subscribed for cvg resource in ha_setup and re0-executed ha_setup. g. Confirmed that after subscribption, cvg online and failed events are sent to Hare f. Created and sent event for disk with resource_status "unknown". Confirmed that event is parsed by health monitor but not sent to Hare

[ArchanaLimaye]

@mukhtar-inamdar

Just 1 query why subscription for CVG is not made as other changes related to CVG you are commiting As per the LLD HA is not going to send and CVG event to Hare. So subscription should not be done for them