search

Seagate / cortx-hare

CORTX Hare configures Motr object store, starts/stops Motr services, and notifies Motr of service and device faults.
https://github.com/Seagate/cortx
Apache License 2.0
13 stars 80 forks source link

CORTX-29700: ha: function to update motr process state is complicated #2039

Closed mssawant closed 2 years ago

mssawant commented 2 years ago

Presently the handler function to update and broadcast motr process state changes is a bit convulated and is difficult to read. Also, hax updates the remote process's state only in case of node failure but not if process is reported as offline.

Solution:

Signed-off-by: Mandar Sawant mandar.sawant@seagate.com

vaibhavparatwar commented 2 years ago

@pavankrishnat / @supriyachavan4398 Lets get this tested throghly for deployments in loop, degraded mode Reads, dg bytecount features.. at least some regression needs to run. @mssawant please add if i am missing some tests here.

vaibhavparatwar commented 2 years ago

@mssawant can you link it to JIRA id?

mssawant commented 2 years ago

15N deployment worked for me for atleast 3-4 times with the changes,

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x2d3 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x3f8 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184  (RC)
    [started]  hax                 0x7200000000000001:0x127        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x12a        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x137        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x144        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x14b        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0x14e        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x15b        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x168        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1630
    [started]  hax                 0x7200000000000001:0x16f        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@22001
    [started]  ioservice           0x7200000000000001:0x172        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21001
    [started]  ioservice           0x7200000000000001:0x17f        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21002
    [started]  confd               0x7200000000000001:0x18c        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x193        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x196        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x1a3        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x1b0        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x1b7        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x1ba        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x1c7        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x1d4        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x1db        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0x1de        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0x1eb        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0x1f8        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x1ff        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x202        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0x20f        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0x21c        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21003
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x221        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw                 0x7200000000000001:0x224        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x229        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw                 0x7200000000000001:0x22c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x231        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw                 0x7200000000000001:0x234        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1630
    [started]  hax                 0x7200000000000001:0x239        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1630@22001
    [started]  rgw                 0x7200000000000001:0x23c        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1630@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x241        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw                 0x7200000000000001:0x244        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x249        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw                 0x7200000000000001:0x24c        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x251        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw                 0x7200000000000001:0x254        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x259        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw                 0x7200000000000001:0x25c        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x261        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw                 0x7200000000000001:0x264        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x269        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw                 0x7200000000000001:0x26c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x271        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw                 0x7200000000000001:0x274        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21501
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [started]  confd               0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21003
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x279        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw                 0x7200000000000001:0x27c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x281        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw                 0x7200000000000001:0x284        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x289        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  rgw                 0x7200000000000001:0x28c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x291        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw                 0x7200000000000001:0x294        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21501
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0x6c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x73         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x76         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x83         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x90         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x97         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x9a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0xa7         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0xb4         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0xbb         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0xbe         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0xcb         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0xd8         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0xdf         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0xe2         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0xef         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0xfc         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x103        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0x106        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0x113        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0x120        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21003
Devices:
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1630
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1630
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]
pavankrishnat commented 2 years ago

@mssawant I tried deploying 5506-custom-ci on 15N and 3N setups.

On 15N

https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/1648/console Still Observed:

  1. hctl status from all pods: Still seeing CORTX-29441, device and Service status are not displaying correctly.

    On Pod cortx-data-ssc-vm-g2-rhev4-3158-79bf855d69-t47gg

                cortx-data-headless-svc-ssc-vm-g2-rhev4-3159
                [offline]  hax                 0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@22001
                [offline]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21001
                [offline]  ioservice           0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21002
                [offline]  confd               0x7200000000000001:0x6c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21003
    
[root@ssc-vm-g2-rhev4-3156 ~]# time for each in `kubectl get pods | grep cortx-data | awk '{print $1}'` ; do echo $each; time kubectl exec -it $each -c cortx-hax -- hctl status -d | grep -iE 'offline|stopped|unknown' ; done
cortx-data-ssc-vm-g2-rhev4-3156-79f45b9697-7qp2g

real 4m41.529s user 0m0.215s sys 0m0.241s cortx-data-ssc-vm-g2-rhev4-3157-779fbcf8bb-mrbtq

real 2m51.597s user 0m0.168s sys 0m0.288s cortx-data-ssc-vm-g2-rhev4-3158-79bf855d69-t47gg [offline] hax 0x7200000000000001:0x4f inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@22001 [offline] ioservice 0x7200000000000001:0x52 inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21001 [offline] ioservice 0x7200000000000001:0x5f inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21002 [offline] confd 0x7200000000000001:0x6c inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21003

real 4m55.134s user 0m0.229s sys 0m0.280s cortx-data-ssc-vm-g2-rhev4-3159-cc986f47d-8sqf7

real 5m20.156s user 0m0.212s sys 0m0.279s cortx-data-ssc-vm-g2-rhev4-3160-7d45866898-lrs78 [offline] /dev/sdh

real 7m10.013s user 0m0.284s sys 0m0.371s cortx-data-ssc-vm-g2-rhev4-3161-9965b649f-55wr6

real 7m2.011s user 0m0.238s sys 0m0.199s cortx-data-ssc-vm-g2-rhev4-3162-76bc856b8-fnvrv

real 4m5.095s user 0m0.848s sys 0m0.631s cortx-data-ssc-vm-g2-rhev4-3163-668478cdcc-4mm9l

real 4m3.399s user 0m0.197s sys 0m0.203s cortx-data-ssc-vm-g2-rhev4-3164-847686bdb-mnhbp

real 2m25.791s user 0m0.166s sys 0m0.135s cortx-data-ssc-vm-g2-rhev4-3165-6479bcc9bf-z2ttw [offline] /dev/sdd [offline] /dev/sde [offline] /dev/sdc

real 3m15.504s user 0m0.150s sys 0m0.146s cortx-data-ssc-vm-g2-rhev4-3166-54b5fdb7b4-cbcrj [offline] /dev/sdd [offline] /dev/sde [offline] /dev/sdc [offline] /dev/sdg [offline] /dev/sdh [offline] /dev/sdf

real 3m58.551s user 0m0.177s sys 0m0.206s cortx-data-ssc-vm-g2-rhev4-3167-6c986cfcbf-m4pjb [offline] /dev/sdd [offline] /dev/sde [offline] /dev/sdc [offline] /dev/sdg [offline] /dev/sdh [offline] /dev/sdf

real 7m26.754s user 0m0.213s sys 0m0.331s cortx-data-ssc-vm-g2-rhev4-3168-74fb6cfb86-jwz42 [offline] /dev/sdd [offline] /dev/sde [offline] /dev/sdc [offline] /dev/sdg [offline] /dev/sdh [offline] /dev/sdf

real 4m38.285s user 0m0.210s sys 0m0.246s cortx-data-ssc-vm-g2-rhev4-3169-58b9db98d9-tqdrb [offline] /dev/sdd [offline] /dev/sde [offline] /dev/sdc [offline] /dev/sdg [offline] /dev/sdh [offline] /dev/sdf

real 2m36.816s user 0m0.170s sys 0m0.216s cortx-data-ssc-vm-g2-rhev4-3170-b5889589c-v5vpt [offline] /dev/sdd [offline] /dev/sde [offline] /dev/sdc [offline] /dev/sdg [offline] /dev/sdh [offline] /dev/sdf

real 6m4.190s user 0m1.049s sys 0m0.860s

real 70m35.106s user 0m4.720s sys 0m4.735s [root@ssc-vm-g2-rhev4-3156 ~]#

**real    70m35.106s**
user    0m4.720s
sys     0m4.735s

**hctl status responds time on each pod is ~5mins**
2. Still seeing CORTX-29138, In dgmode IO operations are failing

[root@ssc-vm-g2-rhev4-3156 ~]# aws s3 ls --endpoint-url http://10.102.180.30:80

Read timeout on endpoint URL: "http://10.102.180.30:80/"

Healthy Cluster: Bytecount

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-3158 /]# hctl status -d Bytecount: critical : 0 damaged : 0 degraded : 0 healthy : 108352852 Data pool:

fid name

0x6f00000000000001:0x2d3 'storage-set-1__sns'

Profile:

fid name: pool(s)

0x7000000000000001:0x3f8 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None

Services:

Degrade cluster by **deleting 12 Nodes** still Bytecount is showing healthy  count

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-3161 /]# time hctl status -d Bytecount: critical : 0 damaged : 0 degraded : 0 healthy : 1048576000 Data pool:

fid name

0x6f00000000000001:0x2d3 'storage-set-1__sns'

Profile:

fid name: pool(s)

0x7000000000000001:0x3f8 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None

# On 3N
https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/1654/console
Still seeing CORTX-29138, In dgmode IO operations are failing

[root@ssc-vm-g2-rhev4-3171 ~]# aws s3 ls --endpoint-url http://10.98.13.26:80

An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: Unknown

mssawant commented 2 years ago

New build (https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/5521) deployment on another 15N cluster was deployed successfully (again barring the HA pod timeout), with no processes or devices being in offline state.

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x2a3 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x3b7 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x127        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x12a        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x137        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x144        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x14b        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0x14e        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0x15b        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0x168        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x16f        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x172        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0x17f        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0x18c        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x193        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0x196        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x1a3        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x1b0        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x1b7        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x1ba        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x1c7        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x1d4        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279  (RC)
    [started]  hax                 0x7200000000000001:0x1db        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0x1de        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0x1eb        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0x1f8        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21003
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x1fd        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  rgw                 0x7200000000000001:0x200        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x205        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw                 0x7200000000000001:0x208        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x20d        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw                 0x7200000000000001:0x210        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x215        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw                 0x7200000000000001:0x218        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x21d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw                 0x7200000000000001:0x220        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x225        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw                 0x7200000000000001:0x228        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x22d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw                 0x7200000000000001:0x230        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x235        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw                 0x7200000000000001:0x238        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x23d        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw                 0x7200000000000001:0x240        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x245        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw                 0x7200000000000001:0x248        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x24d        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw                 0x7200000000000001:0x250        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x255        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw                 0x7200000000000001:0x258        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x25d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw                 0x7200000000000001:0x260        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x265        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw                 0x7200000000000001:0x268        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21501
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x6c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x73         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  ioservice           0x7200000000000001:0x76         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [started]  ioservice           0x7200000000000001:0x83         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [started]  confd               0x7200000000000001:0x90         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x97         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x9a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0xa7         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0xb4         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0xbb         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0xbe         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0xcb         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0xd8         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0xdf         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0xe2         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0xef         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0xfc         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x103        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x106        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x113        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x120        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21003
Devices:
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# exit
[root@ssc-vm-g2-rhev4-1630 ~]# time for each in `kubectl get pods | grep cortx-data | awk '{print $1}'` ; do echo $each; time kubectl exec -it $each -c cortx-hax -- hctl status -d | grep -iE 'offline|stopped|unknown' ; done
cortx-data-ssc-vm-g2-rhev4-1631-54698c97f9-7jtwj

real    3m59.302s
user    0m0.221s
sys     0m0.173s
cortx-data-ssc-vm-g2-rhev4-1632-59474b9858-s2vz5

real    3m0.104s
user    0m0.215s
sys     0m0.203s
cortx-data-ssc-vm-g2-rhev4-1635-9674dfd4f-wskt6

real    2m16.986s
user    0m0.190s
sys     0m0.124s
cortx-data-ssc-vm-g2-rhev4-2237-76bb6c44bb-5hprx

real    5m13.994s
user    0m0.249s
sys     0m0.210s
cortx-data-ssc-vm-g2-rhev4-2238-75f5dc9dd9-kgtf9

real    4m50.334s
user    0m0.244s
sys     0m0.147s
cortx-data-ssc-vm-g3-rhev4-2107-76757fb6f5-6lsrs

real    3m3.177s
user    0m0.211s
sys     0m0.180s
cortx-data-ssc-vm-g3-rhev4-2184-5ff84bdb68-rq6mg

real    3m48.384s
user    0m0.185s
sys     0m0.210s
cortx-data-ssc-vm-g3-rhev4-2198-67cbff88c5-zgmqh

real    2m14.621s
user    0m0.175s
sys     0m0.109s
cortx-data-ssc-vm-g3-rhev4-2278-59c448bd94-j9jbq

real    6m1.103s
user    0m0.233s
sys     0m0.193s
cortx-data-ssc-vm-g3-rhev4-2279-59bd6b66df-fmdhf

real    5m50.892s
user    0m0.206s
sys     0m0.220s
cortx-data-ssc-vm-g3-rhev4-2281-67dfc9bc66-72vh6

real    3m59.085s
user    0m0.216s
sys     0m0.194s
cortx-data-ssc-vm-g3-rhev4-2282-cd5f67765-2bndv

real    4m51.957s
user    0m0.230s
sys     0m0.159s
cortx-data-ssc-vm-g3-rhev4-2283-76f487c944-pr5gw

real    6m14.742s
user    0m0.253s
sys     0m0.210s
cortx-data-ssc-vm-g3-rhev4-2284-655d77f9b5-fh4fr

real    2m46.422s
user    0m0.256s
sys     0m0.179s

real    58m11.440s
user    0m3.341s
sys     0m2.645s
[root@ssc-vm-g2-rhev4-1630 ~]#
mssawant commented 2 years ago

On another 15N setup with build(https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/5521), data and server pod deployment was successful,

cortx-control-6d99bb587-4vns4                        1/1     Running   0          54m
cortx-data-ssc-vm-g2-rhev4-3156-6856b6c7d7-pvf7m     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3157-7bd8758946-86kv9     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3158-7f5b865cf9-hj5jp     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3159-64cc6b5b4-54mpv      4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3160-7b584f8864-gb2p5     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3161-7c9dcf88b9-2kfww     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3162-fcc4d7b-66fkq        4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3163-5495b4874c-xd5t7     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3164-585c7dcb54-b6ht7     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3165-8bd7bdd99-48ztw      4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3166-6d6bcdc58d-ngvdg     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3167-86c8fdd9f5-fgc2s     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3168-954ff96c4-cwxmg      4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3169-7c9db4f879-qd7x9     4/4     Running   0          53m
cortx-data-ssc-vm-g2-rhev4-3170-6b7b77d48c-wjx9j     4/4     Running   0          53m
cortx-ha-75d6564b9f-8twhx                            3/3     Running   0          44m
cortx-server-ssc-vm-g2-rhev4-3156-57bc77447f-p6hzd   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3157-5d7b6bb64c-gm4l8   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3158-5b65c685d5-n6n8d   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3159-677b9975f9-94kl6   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3160-776b594577-9s8nk   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3161-76b494444c-dvdv4   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3162-55d4d678c-fmqf4    2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3163-78f74c49f9-wtgh7   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3164-86dffff75c-jk8wc   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3165-59d9876999-z68zf   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3166-845b6f9549-z6rxl   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3167-68dd67967-fvlp8    2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3168-677497bf49-8mxhq   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3169-7694fdf959-pkjpg   2/2     Running   0          47m
cortx-server-ssc-vm-g2-rhev4-3170-555cf656d5-c5fwh   2/2     Running   0          47m
kafka-0                                              1/1     Running   0          55m
kafka-1                                              1/1     Running   0          55m
kafka-2                                              1/1     Running   0          55m
zookeeper-0                                          1/1     Running   0          56m
zookeeper-1                                          1/1     Running   0          56m
zookeeper-2                                          1/1     Running   0          56m
[root@ssc-vm-g2-rhev4-3156 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-3156-6856b6c7d7-pvf7m ^C
[root@ssc-vm-g2-rhev4-3156 ~]# time for each in `kubectl get pods | grep cortx-data | awk '{print $1}'` ; do echo $each; time kubectl exec -it $each -c cortx-hax -- hctl status -d | grep -iE 'offline|stopped|unknown' ; done
cortx-data-ssc-vm-g2-rhev4-3156-6856b6c7d7-pvf7m

real    2m46.952s
user    0m0.158s
sys     0m0.163s
cortx-data-ssc-vm-g2-rhev4-3157-7bd8758946-86kv9

real    3m19.604s
user    0m0.193s
sys     0m0.281s
cortx-data-ssc-vm-g2-rhev4-3158-7f5b865cf9-hj5jp

real    3m12.497s
user    0m0.182s
sys     0m0.127s
cortx-data-ssc-vm-g2-rhev4-3159-64cc6b5b4-54mpv

real    2m45.803s
user    0m0.171s
sys     0m0.125s
cortx-data-ssc-vm-g2-rhev4-3160-7b584f8864-gb2p5

real    2m26.002s
user    0m0.157s
sys     0m0.174s
cortx-data-ssc-vm-g2-rhev4-3161-7c9dcf88b9-2kfww

real    2m32.699s
user    0m0.171s
sys     0m0.205s
cortx-data-ssc-vm-g2-rhev4-3162-fcc4d7b-66fkq

real    7m1.902s
user    0m0.284s
sys     0m0.182s
cortx-data-ssc-vm-g2-rhev4-3163-5495b4874c-xd5t7

real    3m17.281s
user    0m0.196s
sys     0m0.159s
cortx-data-ssc-vm-g2-rhev4-3164-585c7dcb54-b6ht7

real    2m29.418s
user    0m0.192s
sys     0m0.173s
cortx-data-ssc-vm-g2-rhev4-3165-8bd7bdd99-48ztw

real    2m35.110s
user    0m0.178s
sys     0m0.093s
cortx-data-ssc-vm-g2-rhev4-3166-6d6bcdc58d-ngvdg

real    5m28.385s
user    0m0.207s
sys     0m0.157s
cortx-data-ssc-vm-g2-rhev4-3167-86c8fdd9f5-fgc2s

real    3m14.888s
user    0m0.191s
sys     0m0.163s
cortx-data-ssc-vm-g2-rhev4-3168-954ff96c4-cwxmg

real    6m33.213s
user    0m0.219s
sys     0m0.244s
cortx-data-ssc-vm-g2-rhev4-3169-7c9db4f879-qd7x9

real    5m36.796s
user    0m0.208s
sys     0m0.192s
cortx-data-ssc-vm-g2-rhev4-3170-6b7b77d48c-wjx9j

real    4m27.294s
user    0m0.272s
sys     0m0.223s

real    57m48.038s
user    0m3.159s
sys     0m2.718s
[root@ssc-vm-g2-rhev4-3156 ~]#

cc @pavankrishnat, @vaibhavparatwar

pavankrishnat commented 2 years ago

@mssawant with latest changes created new build (https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/5543) deployment on 3N & 15N cluster was successfully (again barring the HA pod timeout), with no processes or devices being in offline state.


[root@ssc-vm-g2-rhev4-3156 ~]# time for each in `kubectl get pods | grep cortx-data | awk '{print $1}'` ; do echo $each; time kubectl exec -it $each -c cortx-hax -- hctl status -d | grep -iE 'offline|stopped|unknown' ; done
cortx-data-ssc-vm-g2-rhev4-3156-8bf74fcbc-9nqlf

real    2m18.234s
user    0m0.158s
sys     0m0.109s
cortx-data-ssc-vm-g2-rhev4-3157-664577c4b6-w7png

real    2m40.184s
user    0m0.188s
sys     0m0.126s
cortx-data-ssc-vm-g2-rhev4-3158-85dd694ccf-92dt2

real    2m56.717s
user    0m0.209s
sys     0m0.271s
cortx-data-ssc-vm-g2-rhev4-3159-5dcd54bd64-mbnbm

real    5m54.417s
user    0m0.233s
sys     0m0.249s
cortx-data-ssc-vm-g2-rhev4-3160-7dd7765884-lnj6m

real    2m35.177s
user    0m0.167s
sys     0m0.194s
cortx-data-ssc-vm-g2-rhev4-3161-6488ff545c-62qb9

real    4m14.091s
user    0m0.225s
sys     0m0.120s
cortx-data-ssc-vm-g2-rhev4-3162-77ddf6ffd-vnwhw

real    2m15.210s
user    0m0.140s
sys     0m0.151s
cortx-data-ssc-vm-g2-rhev4-3163-d88b896b7-ll6gf

real    2m40.105s
user    0m0.169s
sys     0m0.180s
cortx-data-ssc-vm-g2-rhev4-3164-67d4789fd4-t7tm8

real    3m31.688s
user    0m0.187s
sys     0m0.256s
cortx-data-ssc-vm-g2-rhev4-3165-58848b85c9-grfxx

real    5m21.989s
user    0m0.206s
sys     0m0.166s
cortx-data-ssc-vm-g2-rhev4-3166-56847cdc6d-hfjns

real    2m21.205s
user    0m0.169s
sys     0m0.116s
cortx-data-ssc-vm-g2-rhev4-3167-fb668cb7-b7sgf

real    3m29.088s
user    0m0.175s
sys     0m0.128s
cortx-data-ssc-vm-g2-rhev4-3168-b659bf9c6-g6wmc

real    5m26.464s
user    0m0.227s
sys     0m0.168s
cortx-data-ssc-vm-g2-rhev4-3169-86bd47d59-n5k8z

real    5m7.849s
user    0m0.209s
sys     0m0.164s
cortx-data-ssc-vm-g2-rhev4-3170-79bc9d579-88bn2

real    3m40.615s
user    0m0.211s
sys     0m0.149s

real    54m33.267s
user    0m3.058s
sys     0m2.649s

in dgmode seeing below error similar to CORTX-29138

kubectl logs cortx-server-ssc-vm-g2-rhev4-3156-84f677769c-zxrb2 -c cortx-rgw
motr[00001]:  97a0  ERROR  [net/ip.c:445:m0_net_hostname_to_ip]  gethostbyname err=1 for cortx-data-headless-svc-ssc-vm-g2-rhev4-3167
motr[00001]:  97a0  ERROR  [net/ip.c:447:m0_net_hostname_to_ip]  <! rc=1
motr[00001]:  98d0  ERROR  [net/libfab/libfab.c:2247:libfab_dns_resolve_retry]  gethostbyname() failed with err 1 for cortx-data-headless-svc-ssc-vm-g2-rhev4-3167@21001
motr[00001]:  9b50  ERROR  [rpc/frmops.c:576:item_fail]  packet 0x5649f5437980, item 0x5649f28851a0[233] failed with ri_error=-110
pavankrishnat commented 2 years ago

@mssawant, with custom-ci#5567 build deployed 2 times, Not observed CORTX-29441, all processes or devices being in Online states, but after deleting pod still Observing CORTX-28823 CORTX-29700 and CORTX-29138 hctl status on healthy cluster after deployment:

[root@ssc-vm-g2-rhev4-1630 ~]# time for each in `kubectl get pods | grep cortx-data | awk '{print $1}'` ; do echo $each; time kubectl exec -it $each -c cortx-hax -- hctl status -d | grep -iE 'offline[18/18]
d|unknown' ; done
cortx-data-ssc-vm-g2-rhev4-1631-847c8d8b99-vj6mc

real    3m40.312s
user    0m0.192s
sys     0m0.090s
cortx-data-ssc-vm-g2-rhev4-1632-68fbd868d8-vxkwn

real    4m32.701s
user    0m0.233s
sys     0m0.146s
cortx-data-ssc-vm-g2-rhev4-1635-84f59f9c99-mdw9d

real    3m43.091s
user    0m0.233s
sys     0m0.214s
cortx-data-ssc-vm-g2-rhev4-2237-bcbc8468-pfxqv

real    4m2.190s
user    0m0.189s
sys     0m0.212s
cortx-data-ssc-vm-g2-rhev4-2238-5b98f8bcb5-cwrhs

real    3m34.936s
user    0m0.210s
sys     0m0.167s
cortx-data-ssc-vm-g3-rhev4-2107-79498894c9-zj7q8

real    3m56.776s
user    0m0.240s
sys     0m0.146s
cortx-data-ssc-vm-g3-rhev4-2184-cbdf849f4-cwm6z

real    2m19.567s
user    0m0.190s
sys     0m0.123s
cortx-data-ssc-vm-g3-rhev4-2198-cf9f695d7-q65ms

real    2m10.416s
user    0m0.152s
sys     0m0.097s
cortx-data-ssc-vm-g3-rhev4-2278-64f5958754-rg99g

real    4m25.717s
user    0m0.215s
sys     0m0.142s
cortx-data-ssc-vm-g3-rhev4-2279-56f6b49959-tbvtp

real    2m40.975s
user    0m0.216s
sys     0m0.243s
cortx-data-ssc-vm-g3-rhev4-2281-5554df675d-4t2z5

real    1m33.994s
user    0m0.173s
sys     0m0.110s
cortx-data-ssc-vm-g3-rhev4-2282-df77596c9-xtkhk

real    3m55.266s
user    0m0.211s
sys     0m0.164s
cortx-data-ssc-vm-g3-rhev4-2283-55c69b68d6-c8bml

real    2m43.833s
user    0m0.206s
sys     0m0.168s
cortx-data-ssc-vm-g3-rhev4-2284-fc96cccb5-t99vj

real    3m17.352s
user    0m0.228s
sys     0m0.138s

real    46m37.295s
user    0m3.071s
sys     0m2.240s
[root@ssc-vm-g2-rhev4-1630 ~]#

On Healthy cluster after IO's

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1631-847c8d8b99-vj6mc -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 891988650
Data pool:
    # fid name
    0x6f00000000000001:0x2a3 'storage-set-1__sns'

Deleted data pod in 2 ways

  1. using kubectl delete pod cortx-data-ssc-vm-g3-rhev4-2279-56f6b49959-tbvtp CORTX-28823 immediate after pod deleted new pod got created with different IP, but still hctl status shows all processes offline for that node. 
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
[offline]  hax                 0x7200000000000001:0x127        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
[offline]  ioservice           0x7200000000000001:0x12a        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
[offline]  ioservice           0x7200000000000001:0x137        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
[offline]  confd               0x7200000000000001:0x144        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21003
Devices:
cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
[online]  /dev/sdd
[online]  /dev/sde
[online]  /dev/sdc
[online]  /dev/sdg
[online]  /dev/sdh
[online]  /dev/sdf

    consul kv get -recurse CORTX-29700

    m0conf/nodes/0x6e00000000000001:0x123:{"name": "cortx-data-headless-svc-ssc-vm-g3-rhev4-2279", "state": "M0_NC_ONLINE"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x127:{"name": "hax", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x127/services/0x7300000000000001:0x128:{"name": "ha", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x127/services/0x7300000000000001:0x129:{"name": "rms", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a:{"name": "m0_server", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x12b:{"name": "rms", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x12c:{"name": "ios", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x12c/sdevs/0x6400000000000001:0x12d:{"path": "/dev/sdd", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x12c/sdevs/0x6400000000000001:0x12f:{"path": "/dev/sde", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x131:{"name": "sns_rep", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x132:{"name": "sns_reb", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x133:{"name": "addb2", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x134:{"name": "cas", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x134/sdevs/0x6400000000000001:0x28b:{"path": "/dev/sdc", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x135:{"name": "iscs", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x12a/services/0x7300000000000001:0x136:{"name": "fdmi", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137:{"name": "m0_server", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x138:{"name": "rms", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x139:{"name": "ios", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x139/sdevs/0x6400000000000001:0x13a:{"path": "/dev/sdg", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x139/sdevs/0x6400000000000001:0x13c:{"path": "/dev/sdh", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x13e:{"name": "sns_rep", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x13f:{"name": "sns_reb", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x140:{"name": "addb2", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x141:{"name": "cas", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x141/sdevs/0x6400000000000001:0x28d:{"path": "/dev/sdf", "state": "online"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x142:{"name": "iscs", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x137/services/0x7300000000000001:0x143:{"name": "fdmi", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x144:{"name": "m0_server", "state": "offline"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x144/services/0x7300000000000001:0x145:{"name": "rms", "state": "M0_NC_UNKNOWN"}
m0conf/nodes/0x6e00000000000001:0x123/processes/0x7200000000000001:0x144/services/0x7300000000000001:0x146:{"name": "confd", "state": "M0_NC_UNKNOWN"}
    processes/0x7200000000000001:0x127:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x12a:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x137:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x144:{"state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}

    cat /etc/cortx/log/hare/log/4f522fc010784089b5e3ace3530c3c92/hare-hax.log

    2022-04-01 08:50:25,722 [DEBUG] {ThreadPoolExecutor-0_33} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=OK), HAState(fid=0x7200000000000001:0x113, status=OK)]
2022-04-01 08:50:48,540 [DEBUG] {ThreadPoolExecutor-0_4} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=FAILED), HAState(fid=0x7200000000000001:0x137, status=FAILED), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=OK), HAState(fid=0x7200000000000001:0x113, status=OK)]
2022-04-01 08:51:06,572 [DEBUG] {qconsumer-11} Broadcasting HA states [] over ha_link
2022-04-01 08:51:09,791 [INFO] {qconsumer-12} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 08:53:02,986 [DEBUG] {qconsumer-12} Broadcasting HA states [HAState(fid=0x7200000000000001:0x144, status=OFFLINE), HAState(fid=0x7200000000000001:0x144, status=OFFLINE)] over ha_link
2022-04-01 08:53:22,747 [INFO] {qconsumer-3} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 08:54:37,972 [DEBUG] {qconsumer-3} Broadcasting HA states [HAState(fid=0x7200000000000001:0x144, status=OFFLINE)] over ha_link
2022-04-01 08:54:46,487 [INFO] {qconsumer-13} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 08:57:17,376 [DEBUG] {qconsumer-13} Broadcasting HA states [HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK)] over ha_link
2022-04-01 08:58:01,079 [INFO] {qconsumer-13} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 08:59:57,971 [DEBUG] {qconsumer-13} Broadcasting HA states [] over ha_link
2022-04-01 08:59:58,181 [INFO] {qconsumer-30} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 09:03:54,614 [DEBUG] {qconsumer-30} Broadcasting HA states [] over ha_link
  2. using kubectl delete deployment cortx-data-ssc-vm-g3-rhev4-2284 and kubectl create -f cortx-data-2284.yaml CORTX-29138 After creating pod hctl status took ~10 mins to shows all services and devices states to online. 
    2022-04-01 10:00:57,087 [DEBUG] {qconsumer-22} Broadcasting HA states [HAState(fid=0x7200000000000001:0x120, status=OFFLINE), HAState(fid=0x7200000000000001:0x120, status=OFFLINE)] over ha_link
2022-04-01 10:01:25,842 [DEBUG] {qconsumer-22} Broadcasting 6 notes
2022-04-01 10:03:53,581 [DEBUG] {qconsumer-22} Broadcasting HA states [] over ha_link
2022-04-01 10:19:11,682 [DEBUG] {qconsumer-3} Broadcasting HA states [] over ha_link
2022-04-01 10:22:12,397 [DEBUG] {qconsumer-28} Broadcasting HA states [HAState(fid=0x7200000000000001:0x120, status=OK)] over ha_link
2022-04-01 10:22:43,688 [DEBUG] {qconsumer-28} Broadcasting 7 notes
    2022-04-01 09:55:13,679 [DEBUG] {qconsumer-3} Broadcasting HA states [] over ha_link
2022-04-01 09:55:13,830 [INFO] {qconsumer-3} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 09:57:12,982 [DEBUG] {qconsumer-3} Broadcasting HA states [] over ha_link
2022-04-01 09:57:13,186 [INFO] {qconsumer-22} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 09:58:24,940 [DEBUG] {ThreadPoolExecutor-0_26} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=FAILED), HAState(fid=0x7200000000000001:0x113, status=OK)]
2022-04-01 09:58:26,632 [DEBUG] {ThreadPoolExecutor-0_13} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=FAILED), HAState(fid=0x7200000000000001:0x113, status=FAILED)]
2022-04-01 09:58:56,680 [DEBUG] {qconsumer-22} Broadcasting HA states [] over ha_link
2022-04-01 09:58:57,807 [INFO] {qconsumer-22} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 09:58:58,272 [DEBUG] {ThreadPoolExecutor-0_27} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=FAILED), HAState(fid=0x7200000000000001:0x113, status=FAILED)]
2022-04-01 10:00:57,087 [DEBUG] {qconsumer-22} Broadcasting HA states [HAState(fid=0x7200000000000001:0x120, status=OFFLINE), HAState(fid=0x7200000000000001:0x120, status=OFFLINE)] over ha_link
2022-04-01 10:01:25,907 [INFO] {qconsumer-22} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:03:53,581 [DEBUG] {qconsumer-22} Broadcasting HA states [] over ha_link
2022-04-01 10:03:54,323 [INFO] {qconsumer-22} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:07:38,589 [DEBUG] {qconsumer-22} Broadcasting HA states [] over ha_link
2022-04-01 10:07:39,122 [INFO] {qconsumer-15} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:09:22,581 [DEBUG] {qconsumer-15} Broadcasting HA states [] over ha_link
2022-04-01 10:09:22,890 [INFO] {qconsumer-15} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:11:16,524 [DEBUG] {qconsumer-15} Broadcasting HA states [] over ha_link
2022-04-01 10:11:16,853 [INFO] {qconsumer-15} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:13:16,883 [DEBUG] {ThreadPoolExecutor-0_28} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=FAILED), HAState(fid=0x7200000000000001:0x113, status=FAILED)]
2022-04-01 10:13:21,644 [DEBUG] {ThreadPoolExecutor-0_3} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=FAILED), HAState(fid=0x7200000000000001:0x113, status=OK)]
2022-04-01 10:13:24,739 [DEBUG] {ThreadPoolExecutor-0_29} Reporting ha states: [HAState(fid=0x7200000000000001:0x9a, status=OK), HAState(fid=0x7200000000000001:0xa7, status=OK), HAState(fid=0x7200000000000001:0x1ba, status=OK), HAState(fid=0x7200000000000001:0x1c7, status=OK), HAState(fid=0x7200000000000001:0x196, status=OK), HAState(fid=0x7200000000000001:0x1a3, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xbe, status=OK), HAState(fid=0x7200000000000001:0xcb, status=OK), HAState(fid=0x7200000000000001:0x52, status=OK), HAState(fid=0x7200000000000001:0x5f, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0xa, status=OK), HAState(fid=0x7200000000000001:0x17, status=OK), HAState(fid=0x7200000000000001:0x76, status=OK), HAState(fid=0x7200000000000001:0x83, status=OK), HAState(fid=0x7200000000000001:0x12a, status=OK), HAState(fid=0x7200000000000001:0x137, status=OK), HAState(fid=0x7200000000000001:0x2e, status=OK), HAState(fid=0x7200000000000001:0x3b, status=OK), HAState(fid=0x7200000000000001:0x14e, status=OK), HAState(fid=0x7200000000000001:0x15b, status=OK), HAState(fid=0x7200000000000001:0x1de, status=OK), HAState(fid=0x7200000000000001:0x1eb, status=OK), HAState(fid=0x7200000000000001:0x106, status=OK), HAState(fid=0x7200000000000001:0x113, status=OK)]
2022-04-01 10:15:07,279 [DEBUG] {qconsumer-15} Broadcasting HA states [] over ha_link
2022-04-01 10:15:08,396 [INFO] {qconsumer-3} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:19:11,682 [DEBUG] {qconsumer-3} Broadcasting HA states [] over ha_link
2022-04-01 10:19:13,113 [INFO] {qconsumer-28} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:22:12,397 [DEBUG] {qconsumer-28} Broadcasting HA states [HAState(fid=0x7200000000000001:0x120, status=OK)] over ha_link
2022-04-01 10:22:43,694 [INFO] {qconsumer-28} HA states: [HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
2022-04-01 10:24:46,478 [DEBUG] {qconsumer-28} Broadcasting HA states [] over ha_link
2022-04-01 10:24:47,503 [INFO] {qconsumer-28} HA states: [HAState(fid=0x7200000000000001:0x106, status=OK), HAState(fid=0x7200000000000001:0x120, status=OK), HAState(fid=0x7200000000000001:0x144, status=OK), HAState(fid=0x7200000000000001:0x172, status=OK), HAState(fid=0x7200000000000001:0x17f, status=OK), HAState(fid=0x7200000000000001:0x18c, status=OK), HAState(fid=0x7200000000000001:0x1b0, status=OK), HAState(fid=0x7200000000000001:0x1d4, status=OK), HAState(fid=0x7200000000000001:0x1f8, status=OK), HAState(fid=0x7200000000000001:0x24, status=OK), HAState(fid=0x7200000000000001:0x48, status=OK), HAState(fid=0x7200000000000001:0x6c, status=OK), HAState(fid=0x7200000000000001:0x90, status=OK), HAState(fid=0x7200000000000001:0xb4, status=OK), HAState(fid=0x7200000000000001:0xd8, status=OK), HAState(fid=0x7200000000000001:0xe2, status=OK), HAState(fid=0x7200000000000001:0xef, status=OK), HAState(fid=0x7200000000000001:0xfc, status=OK)]
    2022-04-01 09:51:04,982 [DEBUG] {qconsumer-3} KVGET key=m0conf/nodes, kwargs={'recurse': True}
motr[00012]:  b850  ERROR  [spiel/cmd.c:2235:m0_spiel_proc_counters_fetch]  <! rc=-1
2022-04-01 09:51:05,185 [ERROR] {byte-count-updater} Aborting due to an error
Traceback (most recent call last):
File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/bytecount.py", line 138, in _execute
motr.get_proc_bytecount(ios)
File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 715, in get_proc_bytecount
raise RuntimeError('Bytecount stats unavailable')
RuntimeError: Bytecount stats unavailable
2022-04-01 09:51:05,186 [DEBUG] {byte-count-updater} byte-count updater thread exited
2022-04-01 09:51:05,715 [DEBUG] {qconsumer-10} KVGET key=m0conf/nodes/0x6e00000000000001:0x1b3, kwargs={}
mssawant commented 2 years ago

Pod delete with latest build,

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x2a3 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x3b7 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x127        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0x12a        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0x137        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0x144        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x14b        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0x14e        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x15b        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x168        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x16f        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x172        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x17f        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x18c        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x193        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x196        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x1a3        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x1b0        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x1b7        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x1ba        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x1c7        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x1d4        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x1db        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x1de        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x1eb        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x1f8        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21003
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x1fd        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw                 0x7200000000000001:0x200        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x205        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw                 0x7200000000000001:0x208        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x20d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw                 0x7200000000000001:0x210        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x215        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw                 0x7200000000000001:0x218        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x21d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw                 0x7200000000000001:0x220        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x225        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw                 0x7200000000000001:0x228        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x22d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw                 0x7200000000000001:0x230        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x235        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw                 0x7200000000000001:0x238        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x23d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw                 0x7200000000000001:0x240        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x245        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw                 0x7200000000000001:0x248        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x24d        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw                 0x7200000000000001:0x250        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x255        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw                 0x7200000000000001:0x258        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [offline]  hax                 0x7200000000000001:0x25d        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [offline]  rgw                 0x7200000000000001:0x260        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x265        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw                 0x7200000000000001:0x268        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21501
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [offline]  hax                 0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [offline]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [offline]  ioservice           0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [offline]  confd               0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0x6c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x73         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0x76         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0x83         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0x90         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x97         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x9a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0xa7         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0xb4         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238  (RC)
    [started]  hax                 0x7200000000000001:0xbb         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0xbe         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0xcb         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0xd8         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0xdf         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0xe2         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0xef         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0xfc         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x103        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0x106        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0x113        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0x120        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21003
Devices:
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [offline]  /dev/sdd
    [offline]  /dev/sde
    [offline]  /dev/sdc
    [offline]  /dev/sdg
    [offline]  /dev/sdh
    [offline]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]#
processes/0x7200000000000001:0x103:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x106:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x113:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x120:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x127:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x12a:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x137:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x144:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x14b:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x14e:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x15b:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x168:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x16f:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x17:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x172:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x17f:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x18c:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x193:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x196:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1a3:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1b0:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1b7:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x1ba:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1c7:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1d4:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1db:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x1de:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1eb:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1f8:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x1fd:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x200:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x205:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x208:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x20d:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x210:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x215:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x218:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x21d:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x220:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x225:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x228:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x22d:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x230:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x235:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x238:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x23d:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x24:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x240:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x245:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x248:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x24d:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x250:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x255:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x258:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x25d:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x260:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x265:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x268:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}
processes/0x7200000000000001:0x2b:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x2e:{"state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x3b:{"state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x48:{"state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x4f:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x52:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x5f:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x6c:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x7:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x73:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x76:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x83:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x90:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0x97:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0x9a:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xa:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xa7:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xb4:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xbb:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0xbe:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xcb:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xd8:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xdf:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}
processes/0x7200000000000001:0xe2:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xef:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
processes/0x7200000000000001:0xfc:{"state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}
stats/filesystem:{"stats": {"fs_free_seg": 751636450736, "fs_total_seg": 751643681920, "fs_free_disk": 1503238553600, "fs_avail_disk": 1503238553600, "fs_total_disk": 1503238553600, "fs_svc_total": 52, "fs_svc_replied": 48}, "timestamp": 1648815014.328638, "date": "2022-04-01T12:10:14.328638"}
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]#

Presently processing the broadcast event takes sometime in Hare, due to other priority events from filesystemstats and bytecount threads. This needs to be further optimized in a separate task.