Problem Statement

Change HA conf-store delimiter '/' to '>' to sync it with cortx delimiter and to remove the limitation of '/' use in key name.
As in the case of disk id which is like '/dev/sdc' format cannot store in conf-store as '/' is being used as a delimiter for key parsing.
this task was created to get rid of this limitation during disk-related keys, decided to store as '-dev-sdc' but after this change, our code will be using '>' and cortx also using the same delimiter and this will remove the limitation of use of '/'.

Design

Change HA conf-store delimiter '/' to '>' to sync it with cortx delimiter
confstore (consul kv store) before the change:

Note this command output with old code so you may found some other minor changes in both this point:

root@cortx-control /]# consul kv get -http-addr=consul-server-0.consul-server.default.svc.cluster.local:8500 --recurse cortx/ha/v1
cortx/ha/v1/cluster_cardinality:{"num_nodes": 6, "node_list": ["09ab8f9511fe47a6b736f87bbeee90a3", "afbcd1d0bc2544d9b5928dfd01458c06", "6f04649c0ac640e58cbc37c91b4a85e8", "0b1750ffa59f49dd85f30158a5bfd972", "776cd753bb574799b2dc578d604e7919", "9f658a95c34a448d8d51bd43486fdde1"]}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/health:{"events": [{"event_timestamp": "1647260335", "created_timestamp": "1647260339", "status": "online", "specific_info": null}], "action": {"modified_timestamp": "1647260339", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/health:{"events": [{"event_timestamp": "1647260335", "created_timestamp": "1647260339", "status": "online", "specific_info": null}], "action": {"modified_timestamp": "1647260339", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/health:{"events": [{"event_timestamp": "1647260335", "created_timestamp": "1647260338", "status": "online", "specific_info": null}], "action": {"modified_timestamp": "1647260338", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/node/09ab8f9511fe47a6b736f87bbeee90a3/health:{"events": [{"event_timestamp": "1647260336", "created_timestamp": "1647260341", "status": "online", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0147-6946c9b4f5-wspbl", "pod_restart": 0}}], "action": {"modified_timestamp": "1647260341", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/node/0b1750ffa59f49dd85f30158a5bfd972/health:{"events": [{"event_timestamp": "1647260335", "created_timestamp": "1647260338", "status": "online", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0148-6b95dd8dbf-4gnzn", "pod_restart": 0}}], "action": {"modified_timestamp": "1647260338", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/node/6f04649c0ac640e58cbc37c91b4a85e8/health:{"events": [{"event_timestamp": "1647260336", "created_timestamp": "1647260341", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "pod_restart": 0}}], "action": {"modified_timestamp": "1647260341", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/node/776cd753bb574799b2dc578d604e7919/health:{"events": [{"event_timestamp": "1647260336", "created_timestamp": "1647260340", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0147-5489b5dfd7-4nb9p", "pod_restart": 0}}], "action": {"modified_timestamp": "1647260340", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/node/9f658a95c34a448d8d51bd43486fdde1/health:{"events": [{"event_timestamp": "1647260337", "created_timestamp": "1647260342", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "pod_restart": 0}}], "action": {"modified_timestamp": "1647260342", "status": "pending"}, "attributes": {}}
cortx/ha/v1/cortx/ha/system/cluster/fbfd048121d1486ba781438bf61404dc/site/1/rack/1/node/afbcd1d0bc2544d9b5928dfd01458c06/health:{"events": [{"event_timestamp": "1647260337", "created_timestamp": "1647260342", "status": "online", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "pod_restart": 0}}], "action": {"modified_timestamp": "1647260342", "status": "pending"}, "attributes": {}}
cortx/ha/v1/events/node/failed:["hare"]
cortx/ha/v1/events/node/online:["hare"]
cortx/ha/v1/events/subscribe/hare:["node/online", "node/failed"]
cortx/ha/v1/message_type/hare:ha_event_hare
[root@cortx-control /]# consul kv delete -http-addr=consul-server-0.consul-server.default.svc.cluster.local:8500 --recurse cortx/ha/v1
Success! Deleted keys with prefix: cortx/ha/v1
[root@cortx-control /]# consul kv get -http-addr=consul-server-0.consul-server.default.svc.cluster.local:8500 --recurse cortx/ha/v1

confstore (consul kv store) after the change:

[root@cortx-ha-headless-svc /]# consul kv get -http-addr=consul-server-0.consul-server.default.svc.cluster.local:8500 -recurse "cortx>ha>v1"
cortx>ha>v1:
cortx>ha>v1>action>disk>failed:["publish"]
cortx>ha>v1>action>disk>online:["publish"]
cortx>ha>v1>action>node>failed:["publish"]
cortx>ha>v1>action>node>online:["publish"]
cortx>ha>v1>cluster_cardinality:{"num_nodes": 6, "node_list": ["0b1750ffa59f49dd85f30158a5bfd972", "776cd753bb574799b2dc578d604e7919", "9f658a95c34a448d8d51bd43486fdde1", "09ab8f9511fe47a6b736f87bbeee90a3", "afbcd1d0bc2544d9b5928dfd01458c06", "6f04649c0ac640e58cbc37c91b4a85e8"]}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>health:{"events": [{"event_timestamp": "1647277592.7543564", "created_timestamp": "1647277594", "status": "online", "specific_info": null}, {"event_timestamp": "1647277585", "created_timestamp": "1647277587", "status": "degraded", "specific_info": null}], "action": {"modified_timestamp": "1647277594", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>health:{"events": [{"event_timestamp": "1647277592.7543564", "created_timestamp": "1647277594", "status": "online", "specific_info": null}, {"event_timestamp": "1647277585", "created_timestamp": "1647277587", "status": "degraded", "specific_info": null}], "action": {"modified_timestamp": "1647277594", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>health:{"events": [{"event_timestamp": "1647277592.7543564", "created_timestamp": "1647277593", "status": "online", "specific_info": null}, {"event_timestamp": "1647277585", "created_timestamp": "1647277587", "status": "degraded", "specific_info": null}], "action": {"modified_timestamp": "1647277593", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>09ab8f9511fe47a6b736f87bbeee90a3>health:{"events": [{"event_timestamp": "1647277587.8196998", "created_timestamp": "1647277588", "status": "online", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0147-6946c9b4f5-wspbl", "is_status": "True"}}, {"event_timestamp": "1647277588", "created_timestamp": "1647277588", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647277588", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>0b1750ffa59f49dd85f30158a5bfd972>health:{"events": [{"event_timestamp": "1647277587.436991", "created_timestamp": "1647277587", "status": "online", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0148-6b95dd8dbf-4gnzn", "is_status": "True"}}, {"event_timestamp": "1647277585", "created_timestamp": "1647277585", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647277587", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>6f04649c0ac640e58cbc37c91b4a85e8>health:{"events": [{"event_timestamp": "1647277592.7543564", "created_timestamp": "1647277593", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}, {"event_timestamp": "1647277589", "created_timestamp": "1647277589", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647277593", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>776cd753bb574799b2dc578d604e7919>health:{"events": [{"event_timestamp": "1647277587.6509604", "created_timestamp": "1647277588", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0147-5489b5dfd7-4nb9p", "is_status": "True"}}, {"event_timestamp": "1647277587", "created_timestamp": "1647277588", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647277588", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>9f658a95c34a448d8d51bd43486fdde1>health:{"events": [{"event_timestamp": "1647277592.4723332", "created_timestamp": "1647277592", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}, {"event_timestamp": "1647277588", "created_timestamp": "1647277588", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647277592", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>afbcd1d0bc2544d9b5928dfd01458c06>health:{"events": [{"event_timestamp": "1647277588.1606376", "created_timestamp": "1647277589", "status": "online", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}, {"event_timestamp": "1647277589", "created_timestamp": "1647277589", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647277589", "status": "pending"}, "attributes": {}}
cortx>ha>v1>events>disk>failed:["hare"]
cortx>ha>v1>events>disk>online:["hare"]
cortx>ha>v1>events>node>failed:["hare"]
cortx>ha>v1>events>node>online:["hare"]
cortx>ha>v1>events>subscribe>hare:["node>online", "node>failed", "disk>online", "disk>failed"]
cortx>ha>v1>message_type>hare:ha_event_hare
[root@cortx-ha-headless-svc /]#

Coding

[x] Coding conventions are followed and code is consistent

Testing

[ ] Unit and System Tests are added
[x] Test Cases cover Happy Path, Non-Happy Path and Scalability
[x] Testing was performed with RPM

Test scenarios:

Test the cleanup and config command see above output of consul kv store get after execution of config
Check pod delete (Node restart) case consul is getting updated successfully see above output

check the subscription of the event, The event is being sent to hare see below logs of health monitor

2022-03-14 17:33:46 health_monitor [38]: INFO [publish] Sending action event {"header": {"version": "1.0", "timestamp": "1647279226.7627716", "event_id": "1647279226.76277161f03777460864d0980271d932e7e57b4"}, "payload": {"source": "", "cluster_id": "fbfd048121d1486ba781438bf61404dc", "site_id": "0xBAADFOOD", "rack_id": "0xBAADFOOD", "storageset_id": "776cd753bb574799b2dc578d604e7919", "node_id": "776cd753bb574799b2dc578d604e7919", "resource_type": "node", "resource_id": "776cd753bb574799b2dc578d604e7919", "resource_status": "failed", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0147-5489b5dfd7-4nb9p"}}} to component hare
2022-03-14 17:33:47 health_monitor [38]: INFO [evaluate] Evaluated action [] for key action>rack>threshold_breached:low
2022-03-14 17:33:48 health_monitor [38]: INFO [evaluate] Evaluated action [] for key action>site>threshold_breached:low
2022-03-14 17:33:49 health_monitor [38]: INFO [evaluate] Evaluated action [] for key action>cluster>threshold_breached:low
2022-03-14 17:33:50 health_monitor [38]: INFO [evaluate] Evaluated action ['publish'] for key action>node>online
2022-03-14 17:33:50 health_monitor [38]: INFO [process_event] Evaluated {"source": "", "event_id": "1647279229.46264788084ed3465344d284101ca7543cd8a7", "event_type": "online", "severity": "informational", "site_id": "0xBAADFOOD", "rack_id": "0xBAADFOOD", "cluster_id": "fbfd048121d1486ba781438bf61404dc", "storageset_id": "776cd753bb574799b2dc578d604e7919", "node_id": "776cd753bb574799b2dc578d604e7919", "host_id": "776cd753bb574799b2dc578d604e7919", "resource_type": "node", "timestamp": "1647279229.462647", "resource_id": "776cd753bb574799b2dc578d604e7919", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0147-5489b5dfd7-sr4hc"}} with action ['publish']
2022-03-14 17:33:50 health_monitor [38]: INFO [act] Default action handler with Event: 1647279229.46264788084ed3465344d284101ca7543cd8a7 and actions : ['publish']
2022-03-14 17:33:50 health_monitor [38]: INFO [publish] Sending action event {"header": {"version": "1.0", "timestamp": "1647279230.561354", "event_id": "1647279230.561354d01c19cb23054c75825328a9316c7d87"}, "payload": {"source": "", "cluster_id": "fbfd048121d1486ba781438bf61404dc", "site_id": "0xBAADFOOD", "rack_id": "0xBAADFOOD", "storageset_id": "776cd753bb574799b2dc578d604e7919", "node_id": "776cd753bb574799b2dc578d604e7919", "resource_type": "node", "resource_id": "776cd753bb574799b2dc578d604e7919", "resource_status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0147-5489b5dfd7-sr4hc"}}} to component hare
2022-03-14 17:33:52 health_monitor [38]: INFO [evaluate] Evaluated action [] for key action>rack>fault_resolved
2022-03-14 17:33:52 health_monitor [38]: INFO [evaluate] Evaluated action [] for key action>site>fault_resolved
2022-03-14 17:33:53 health_monitor [38]: INFO [evaluate] Evaluated action [] for key action>cluster>fault_resolved

Tested for disk events:

Below is the health system confstore

[root@cortx-ha-headless-svc /]# consul kv get -http-addr=consul-server-0.consul-server.default.svc.cluster.local:8500 -recurse "cortx>ha>v1"
cortx>ha>v1:
cortx>ha>v1>action>disk>failed:["publish"]
cortx>ha>v1>action>disk>online:["publish"]
cortx>ha>v1>action>node>failed:["publish"]
cortx>ha>v1>action>node>online:["publish"]
cortx>ha>v1>cluster_cardinality:{"num_nodes": 6, "node_list": ["0b1750ffa59f49dd85f30158a5bfd972", "09ab8f9511fe47a6b736f87bbeee90a3", "9f658a95c34a448d8d51bd43486fdde1", "afbcd1d0bc2544d9b5928dfd01458c06", "6f04649c0ac640e58cbc37c91b4a85e8", "776cd753bb574799b2dc578d604e7919"]}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>health:{"events": [{"event_timestamp": "1647416828", "created_timestamp": "1647416828", "status": "degraded", "specific_info": null}], "action": {"modified_timestamp": "1647416828", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>health:{"events": [{"event_timestamp": "1647416828", "created_timestamp": "1647416828", "status": "degraded", "specific_info": null}], "action": {"modified_timestamp": "1647416828", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>health:{"events": [{"event_timestamp": "1647416828", "created_timestamp": "1647416828", "status": "degraded", "specific_info": null}], "action": {"modified_timestamp": "1647416828", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>09ab8f9511fe47a6b736f87bbeee90a3>health:{"events": [{"event_timestamp": "1647416828", "created_timestamp": "1647416828", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647416828", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>0b1750ffa59f49dd85f30158a5bfd972>health:{"events": [{"event_timestamp": "1647416828", "created_timestamp": "1647416828", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647416828", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>6f04649c0ac640e58cbc37c91b4a85e8>cvg>cvg-1>disk>/dev/sdc>health:{"events": [{"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428159", "status": "rebalancing", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}, {"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428158", "status": "repaired", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}], "action": {"modified_timestamp": "1647428159", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>6f04649c0ac640e58cbc37c91b4a85e8>cvg>cvg-1>disk>/dev/sdd>health:{"events": [{"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428167", "status": "rebalancing", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}, {"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428166", "status": "repaired", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}], "action": {"modified_timestamp": "1647428167", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>6f04649c0ac640e58cbc37c91b4a85e8>cvg>cvg-1>disk>/dev/sde>health:{"events": [{"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428175", "status": "rebalancing", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}, {"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428174", "status": "repaired", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}], "action": {"modified_timestamp": "1647428175", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>6f04649c0ac640e58cbc37c91b4a85e8>cvg>cvg-1>disk>/dev/sdf>health:{"events": [{"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428183", "status": "rebalancing", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}, {"event_timestamp": "1647428075.1375709", "created_timestamp": "1647428182", "status": "repaired", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0148-59f8cc886b-lq679", "is_status": "True"}}], "action": {"modified_timestamp": "1647428183", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>6f04649c0ac640e58cbc37c91b4a85e8>health:{"events": [{"event_timestamp": "1647416829", "created_timestamp": "1647416829", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647416829", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>776cd753bb574799b2dc578d604e7919>health:{"events": [{"event_timestamp": "1647416830", "created_timestamp": "1647416830", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647416830", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>9f658a95c34a448d8d51bd43486fdde1>cvg>cvg-1>disk>/dev/sdd>health:{"events": [{"event_timestamp": "1647421042.8167903", "created_timestamp": "1647426049", "status": "rebalancing", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}, {"event_timestamp": "1647421042.8167903", "created_timestamp": "1647426049", "status": "repaired", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}], "action": {"modified_timestamp": "1647426049", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>9f658a95c34a448d8d51bd43486fdde1>cvg>cvg-1>disk>/dev/sde>health:{"events": [{"event_timestamp": "1647421042.8167903", "created_timestamp": "1647426055", "status": "rebalancing", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}, {"event_timestamp": "1647421042.8167903", "created_timestamp": "1647426054", "status": "repaired", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}], "action": {"modified_timestamp": "1647426055", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>9f658a95c34a448d8d51bd43486fdde1>cvg>cvg-1>disk>/dev/sdf>health:{"events": [{"event_timestamp": "1647421042.8167903", "created_timestamp": "1647426060", "status": "online", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}, {"event_timestamp": "1647421042.8167903", "created_timestamp": "1647426059", "status": "failed", "specific_info": {"generation_id": "cortx-data-ssc-vm-g4-rhev4-0146-844c7bd59d-5w8tc", "is_status": "True"}}], "action": {"modified_timestamp": "1647426060", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>9f658a95c34a448d8d51bd43486fdde1>health:{"events": [{"event_timestamp": "1647416829", "created_timestamp": "1647416829", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647416829", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>afbcd1d0bc2544d9b5928dfd01458c06>cvg>cvg-1>disk>/dev/sdc>health:{"events": [{"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428062", "status": "rebalancing", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}, {"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428062", "status": "repaired", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}], "action": {"modified_timestamp": "1647428062", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>afbcd1d0bc2544d9b5928dfd01458c06>cvg>cvg-1>disk>/dev/sdd>health:{"events": [{"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428064", "status": "rebalancing", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}, {"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428064", "status": "repaired", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}], "action": {"modified_timestamp": "1647428064", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>afbcd1d0bc2544d9b5928dfd01458c06>cvg>cvg-1>disk>/dev/sde>health:{"events": [{"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428072", "status": "rebalancing", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}, {"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428071", "status": "repaired", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}], "action": {"modified_timestamp": "1647428072", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>afbcd1d0bc2544d9b5928dfd01458c06>cvg>cvg-1>disk>/dev/sdf>health:{"events": [{"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428043", "status": "rebalancing", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}, {"event_timestamp": "1647426061.9213154", "created_timestamp": "1647428042", "status": "repaired", "specific_info": {"generation_id": "cortx-server-ssc-vm-g4-rhev4-0146-568ffd8b4f-svlnk", "is_status": "True"}}], "action": {"modified_timestamp": "1647428043", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>afbcd1d0bc2544d9b5928dfd01458c06>health:{"events": [{"event_timestamp": "1647416829", "created_timestamp": "1647416829", "status": "unknown", "specific_info": null}], "action": {"modified_timestamp": "1647416829", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>ssc-vm-g4-rhev4-0146.colo.seagate.com>cvg>cvg-1>disk>/dev/sdc>health:{"events": [{"event_timestamp": "1647428075.11632", "created_timestamp": "1647428159", "status": "rebalancing", "specific_info": {"is_status": "True"}}, {"event_timestamp": "1647428075.11632", "created_timestamp": "1647428158", "status": "repaired", "specific_info": {"is_status": "True"}}], "action": {"modified_timestamp": "1647428159", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>ssc-vm-g4-rhev4-0146.colo.seagate.com>cvg>cvg-1>disk>/dev/sdd>health:{"events": [{"event_timestamp": "1647428075.11632", "created_timestamp": "1647428167", "status": "rebalancing", "specific_info": {"is_status": "True"}}, {"event_timestamp": "1647428075.11632", "created_timestamp": "1647428166", "status": "repaired", "specific_info": {"is_status": "True"}}], "action": {"modified_timestamp": "1647428167", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>ssc-vm-g4-rhev4-0146.colo.seagate.com>cvg>cvg-1>disk>/dev/sde>health:{"events": [{"event_timestamp": "1647428075.11632", "created_timestamp": "1647428175", "status": "rebalancing", "specific_info": {"is_status": "True"}}, {"event_timestamp": "1647428075.11632", "created_timestamp": "1647428174", "status": "repaired", "specific_info": {"is_status": "True"}}], "action": {"modified_timestamp": "1647428175", "status": "pending"}, "attributes": {}}
cortx>ha>v1>cortx>ha>system>cluster>fbfd048121d1486ba781438bf61404dc>site>0xBAADFOOD>rack>0xBAADFOOD>node>ssc-vm-g4-rhev4-0146.colo.seagate.com>cvg>cvg-1>disk>/dev/sdf>health:{"events": [{"event_timestamp": "1647428075.11632", "created_timestamp": "1647428183", "status": "rebalancing", "specific_info": {"is_status": "True"}}, {"event_timestamp": "1647428075.11632", "created_timestamp": "1647428182", "status": "repaired", "specific_info": {"is_status": "True"}}], "action": {"modified_timestamp": "1647428183", "status": "pending"}, "attributes": {}}
cortx>ha>v1>events>disk>failed:["hare"]
cortx>ha>v1>events>disk>online:["hare"]
cortx>ha>v1>events>node>failed:["hare"]
cortx>ha>v1>events>node>online:["hare"]
cortx>ha>v1>events>subscribe>hare:["node>online", "node>failed", "disk>online", "disk>failed"]
cortx>ha>v1>message_type>hare:ha_event_hare

Review Checklist

[x] PR is self reviewed
[x] JIRA number/GitHub Issue added to PR
[x] Jira and state/status is updated and JIRA is updated with PR link
[x] Check if the description is clear and explained
[x] Is there a change in filename/package/module or signature? [Y/N]:
[x] If yes for above point, is a notification sent to all other cortx components? [Y/N] N
[x] Side effects on other features (deployment/upgrade)? [Y/N] N
[x] Dependencies on other component(s)? [Y/N] N
If yes for above point, post link to the corresponding PR.

Review Checklist

[ ] Is perfline test run and the report with and without the changes updated in the PR? [Y/N]:

Documentation

Checklist for Author

[] Changes done to WIKI / Confluence page / Quick Start Guide
LLD for same is updated

Seagate / cortx-ha

CORTX-29176 : Change HA confstore delimiter '/' to '>' #666

Problem Statement

Design

Coding

Testing

Test scenarios:

Review Checklist

Review Checklist

Documentation