search

Seagate / cortx-hare

CORTX Hare configures Motr object store, starts/stops Motr services, and notifies Motr of service and device faults.
https://github.com/Seagate/cortx
Apache License 2.0
13 stars 80 forks source link

CORTX-33628 : Hax motr node restart fix #2132

Closed mssawant closed 2 years ago

mssawant commented 2 years ago 
CORTX-33628: Hare ha events are not ordered
Hare monitors motr and rgw processes for failures and
broadcasts the same to the entire cluster. But presently
they are not ordered and there's not way to determine the
order in which they were processed across the cluster.
Out of order events affects IO post or during failures
as events are not delivered properly.
Worst, it is extremely difficult to debug the same through
several logs.
Thus, it is important to maintain the event order and logging.

Solution:
- Use broadcast queue to broadcast events and maintain order.
- Let RC only handle the node failure events.
- Let local node handle the process failure event.
- Increase consul service watcher interval and set timeout
  for probe to avoid frequent Consul watcher events.
- Guard from delayed Consul events by checking process
  node availability and allowing process local node to
  process the event if the corresponding node's alive.
mssawant commented 2 years ago

retest this please

vaibhavparatwar commented 2 years ago

@mssawant Please link this with some JIRA ID

supriyachavan4398 commented 2 years ago

Created Custom build at https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7125/console Started (1N master+ 27N worker) deployment at https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/8631/ cc. @mssawant, @vaibhavparatwar

supriyachavan4398 commented 2 years ago

Created custom build with reverted rgw-integration branch: https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7137/parameters/ i.e it with 10 mins time interval. Did 27N deployment with configuration SNS: 10+5+0, DIX: 1+11+0 at https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/8675/ Data pods were generated successfully and Server pods were restarted for the last 2 hours.

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS             RESTARTS         AGE
cortx-consul-client-2dt7r                            1/1     Running            0                4h13m
cortx-consul-client-2zp7b                            1/1     Running            0                4h13m
cortx-consul-client-82znr                            1/1     Running            0                4h13m
cortx-consul-client-84rvm                            1/1     Running            0                4h13m
cortx-consul-client-99v82                            1/1     Running            0                4h14m
cortx-consul-client-bdb4d                            1/1     Running            0                4h14m
cortx-consul-client-dgr8p                            1/1     Running            0                4h13m
cortx-consul-client-dkffh                            1/1     Running            0                4h13m
cortx-consul-client-dscf5                            1/1     Running            0                4h13m
cortx-consul-client-g47bx                            1/1     Running            0                4h13m
cortx-consul-client-g5tg8                            1/1     Running            0                4h13m
cortx-consul-client-gbdqc                            1/1     Running            0                4h13m
cortx-consul-client-gm8rm                            1/1     Running            0                4h13m
cortx-consul-client-gngtb                            1/1     Running            0                4h14m
cortx-consul-client-j4l6c                            1/1     Running            0                4h13m
cortx-consul-client-jbf7r                            1/1     Running            0                4h14m
cortx-consul-client-jjbs5                            1/1     Running            0                4h13m
cortx-consul-client-lz8mn                            1/1     Running            0                4h13m
cortx-consul-client-ng2vm                            1/1     Running            0                4h13m
cortx-consul-client-npwq4                            1/1     Running            0                4h13m
cortx-consul-client-r86ns                            1/1     Running            0                4h13m
cortx-consul-client-smnrt                            1/1     Running            0                4h13m
cortx-consul-client-sw64h                            1/1     Running            0                4h13m
cortx-consul-client-t8cwd                            1/1     Running            0                4h14m
cortx-consul-client-vzkrk                            1/1     Running            0                4h14m
cortx-consul-client-z4ghb                            1/1     Running            0                4h13m
cortx-consul-client-zfghc                            1/1     Running            0                4h13m
cortx-consul-server-0                                1/1     Running            0                4h13m
cortx-consul-server-1                                1/1     Running            4 (6m53s ago)    4h14m
cortx-consul-server-2                                1/1     Running            3 (21m ago)      4h14m
cortx-control-79f99d5cd-fl72l                        1/1     Running            0                4h12m
cortx-data-ssc-vm-g2-rhev4-1631-57fdb87bf5-pzwd8     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-1632-9f74b66db-s87cg      4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-1635-7bcb7d6455-pgp6t     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-2237-5599867558-g25qz     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-2238-5dcf5fcdd7-l2vq6     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3031-66499bf77f-drtp5     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3156-7545d78575-fjfqf     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3157-57b8644f6d-6t92g     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3158-5dffb75557-f4pxb     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3159-5b6d47d5f4-5vth6     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3160-6ccdc8746d-4gsnc     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3166-fd9fcbcd-xkpzq       4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3167-7999b99bcc-5xlbg     4/4     Running            0                4h11m
cortx-data-ssc-vm-g2-rhev4-3170-5657648997-8qwj7     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2107-7cdc84f7c5-qlcbk     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2184-5cddb956bb-ppbrn     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2198-6d976f744c-9mnz4     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2278-6ff69c8b68-pl8rh     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2279-8675687c99-kcrfg     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2281-755765bd9b-gd6w7     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2282-64fc4bc777-t2ggg     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2283-547ffd5fd6-tn6zv     4/4     Running            0                4h11m
cortx-data-ssc-vm-g3-rhev4-2284-d68f975-w6jzh        4/4     Running            0                4h11m
cortx-data-ssc-vm-g4-rhev4-1583-5ccd57f7bc-ptsdl     4/4     Running            0                4h11m
cortx-data-ssc-vm-g4-rhev4-1590-559d4558c-vksnk      4/4     Running            0                4h11m
cortx-data-ssc-vm-g4-rhev4-1591-66d7d7696d-j5cxr     4/4     Running            0                4h11m
cortx-data-ssc-vm-g4-rhev4-1592-848597cb5c-lwnk7     4/4     Running            0                4h11m
cortx-ha-86c7bb66bf-4rcgz                            3/3     Running            0                114m
cortx-kafka-0                                        1/1     Running            1 (4h15m ago)    4h16m
cortx-kafka-1                                        1/1     Running            0                4h16m
cortx-kafka-2                                        1/1     Running            0                4h16m
cortx-server-ssc-vm-g2-rhev4-1631-549c8d7888-wnntw   1/2     Error              17 (6m36s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-1632-54649c4f46-pmrqt   1/2     CrashLoopBackOff   17 (19s ago)     4h7m
cortx-server-ssc-vm-g2-rhev4-1635-6f9cc594d4-vqb52   2/2     Running            17 (5m42s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-2237-c5dbd78fd-wptlb    1/2     CrashLoopBackOff   17 (17s ago)     4h7m
cortx-server-ssc-vm-g2-rhev4-2238-cbbc6884-px827     2/2     Running            17 (6m5s ago)    4h7m
cortx-server-ssc-vm-g2-rhev4-3031-54c98b5ff6-6txps   2/2     Running            17 (6m25s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3156-654bf74cf6-vhhq2   2/2     Running            17 (6m16s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3157-768d8ff798-94twz   2/2     Running            17 (6m22s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3158-78877455fb-wd5hr   2/2     Running            17 (6m16s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3159-65cc8c49f4-m4tn4   2/2     Running            17 (6m27s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3160-875886d6b-k5kfn    2/2     Running            17 (6m16s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3166-f494cd478-n27m9    1/2     CrashLoopBackOff   17 (32s ago)     4h7m
cortx-server-ssc-vm-g2-rhev4-3167-596768fc64-g7rlw   2/2     Running            17 (6m23s ago)   4h7m
cortx-server-ssc-vm-g2-rhev4-3170-5978cbdd4d-mdvr2   2/2     Running            17 (5m57s ago)   4h7m
cortx-server-ssc-vm-g3-rhev4-2107-569c5746cb-89nzb   1/2     CrashLoopBackOff   17 (23s ago)     4h7m
cortx-server-ssc-vm-g3-rhev4-2184-6c9f9cd6-stcjs     1/2     CrashLoopBackOff   17 (27s ago)     4h7m
cortx-server-ssc-vm-g3-rhev4-2198-84c7f7dd8-b6qqv    2/2     Running            17 (6m27s ago)   4h7m
cortx-server-ssc-vm-g3-rhev4-2278-7b6c4c84f4-ljdn4   1/2     Error              17 (6m42s ago)   4h7m
cortx-server-ssc-vm-g3-rhev4-2279-7fc99d5cf6-5sh2k   1/2     CrashLoopBackOff   17 (20s ago)     4h7m
cortx-server-ssc-vm-g3-rhev4-2281-65dcf7d486-wjxvf   1/2     Error              17 (6m37s ago)   4h7m
cortx-server-ssc-vm-g3-rhev4-2282-67fb6fbcb-mg4m7    1/2     CrashLoopBackOff   17 (21s ago)     4h7m
cortx-server-ssc-vm-g3-rhev4-2283-799645bb66-b426r   2/2     Running            17 (6m20s ago)   4h7m
cortx-server-ssc-vm-g3-rhev4-2284-c587964fb-tzjkc    2/2     Running            17 (5m58s ago)   4h7m
cortx-server-ssc-vm-g4-rhev4-1583-7645c4455b-zvfmh   1/2     CrashLoopBackOff   17 (14s ago)     4h7m
cortx-server-ssc-vm-g4-rhev4-1590-8487979dbd-h8jqw   1/2     Error              17 (6m40s ago)   4h7m
cortx-server-ssc-vm-g4-rhev4-1591-57cc855c8b-frzzp   2/2     Running            17 (6m26s ago)   4h7m
cortx-server-ssc-vm-g4-rhev4-1592-849ccffc44-jqdvm   2/2     Running            17 (6m16s ago)   4h7m
cortx-zookeeper-0                                    1/1     Running            0                4h16m
cortx-zookeeper-1                                    1/1     Running            0                4h16m
cortx-zookeeper-2                                    1/1     Running            0                4h16m

Found HAConsistencyException in hare-hax logs of one of the data pods:

2022-07-08 07:50:20,708 [ERROR] {ThreadPoolExecutor-0_64} Failed to process a message #67. The message is skipped.
Traceback (most recent call last):
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 326, in get_services
    return self.cns.catalog.service(service=svc_name)[1]
  File "/opt/seagate/cortx/hare/lib/python3.6/site-packages/consul/base.py", line 1467, in service
    params=params)
  File "/opt/seagate/cortx/hare/lib/python3.6/site-packages/consul/std.py", line 22, in get
    self.session.get(uri, verify=self.verify, cert=self.cert)))
  File "/opt/seagate/cortx/hare/lib/python3.6/site-packages/consul/base.py", line 223, in cb
    CB._status(response, allow_404=allow_404)
  File "/opt/seagate/cortx/hare/lib/python3.6/site-packages/consul/base.py", line 188, in _status
    raise ConsulException("%d %s" % (response.code, response.body))
consul.base.ConsulException: 500 rpc error making call: Raft leader not found in server lookup mapping

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/queue/__init__.py", line 39, in process
    self.payload_process(msg)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/queue/__init__.py", line 67, in payload_process
    handlers[msg_type](payload)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/queue/__init__.py", line 86, in handle_process_state_update
    self.motr.broadcast_ha_states([hastate])
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/consul/cache.py", line 85, in wrapper
    ret_value = f(*args, **kwds)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 316, in broadcast_ha_states
    hax_fid = self.consul_util.get_hax_fid()
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/consul/cache.py", line 128, in wrapper
    ret_value = f(*args, **kwds)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 434, in get_hax_fid
    svc: Optional[Dict[str, Any]] = self._local_service_by_name('hax')
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/consul/cache.py", line 128, in wrapper
    ret_value = f(*args, **kwds)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 417, in _local_service_by_name
    return self._service_by_name(local_nodename, name)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 381, in _service_by_name
    for svc in cat.get_services(svc_name):
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 329, in get_services
    'Could not access Consul Catalog') from e
hax.exception.HAConsistencyException
2022-07-08 07:50:20,710 [DEBUG] {ThreadPoolExecutor-0_64} Message #67 processed
2022-07-08 07:50:20,710 [DEBUG] {ThreadPoolExecutor-0_64} Marking epoch 67 as read
2022-07-08 07:50:20,714 [DEBUG] {ThreadPoolExecutor-0_64} Got HAConsistencyException: Failed to put value to KV while invoking function mark_last_read (attempt 1). The attempt will be repeated in 5 seconds

There is one warning for connection pool size:

2022-07-08 07:46:46,903 [WARNING] {ThreadPoolExecutor-0_47} Connection pool is full, discarding connection: 127.0.0.1. Connection pool size: 10
2022-07-08 07:46:46,910 [WARNING] {ThreadPoolExecutor-0_31} Connection pool is full, discarding connection: 127.0.0.1. Connection pool size: 10
motr[00049]:  8e90   WARN  [ha/halon/interface.c:1119:m0_halon_interface_send]  hl=0x7fab240237f0 ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22002 epoch=0 tag=405 type=2
motr[00049]:  58c0   WARN  [ha/halon/interface.c:433:halon_interface_msg_is_delivered_cb]  hl=0x7fab44010f80 ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001 tag=391
2022-07-08 07:46:46,912 [WARNING] {rconfc-starter} Connection pool is full, discarding connection: 127.0.0.1. Connection pool size: 10
motr[00049]:  8e90   WARN  [ha/halon/interface.c:1119:m0_halon_interface_send]  hl=0x7faacc01eee0 ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001 epoch=0 tag=407 type=2

cc. @mssawant, @vaibhavparatwar

mssawant commented 2 years ago

retest this please

mssawant commented 2 years ago

retest this please

madhavemuri commented 2 years ago

https://jts.seagate.com/browse/CORTX-31257 This issue could be resolved by this PR.

mssawant commented 2 years ago

Test 11N deployment, degraded IO with Node failure and Node restart,

Node failure and degraded write test

[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-rhev4-2451
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1588
    [started]  hax                 0x7200000000000001:0x36         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1588@22001
    [started]  rgw_s3              0x7200000000000001:0x37         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1588@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  rgw_s3              0x7200000000000001:0x39         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x3a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  rgw_s3              0x7200000000000001:0x3b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x3c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  rgw_s3              0x7200000000000001:0x3d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x3e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  rgw_s3              0x7200000000000001:0x3f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1587
    [started]  hax                 0x7200000000000001:0x40         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1587@22001
    [started]  rgw_s3              0x7200000000000001:0x41         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1587@21001
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1587
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1588
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22002
    cortx-data-headless-svc-ssc-vm-rhev4-2450
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1589  (RC)
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@22001
    [started]  ioservice           0x7200000000000001:0x29         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@21002
    [started]  confd               0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  rgw_s3              0x7200000000000001:0x2d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2450
    [started]  hax                 0x7200000000000001:0x2e         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2450@22001
    [started]  rgw_s3              0x7200000000000001:0x2f         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2450@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2451
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2451@22001
    [started]  rgw_s3              0x7200000000000001:0x31         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2451@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x32         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  rgw_s3              0x7200000000000001:0x33         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1589
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1589@22001
    [started]  rgw_s3              0x7200000000000001:0x35         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1589@21001
Devices:
    cortx-data-headless-svc-ssc-vm-rhev4-2451
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1588
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1587
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1587
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1588
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-rhev4-2450
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712

[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 mb s3://test
make_bucket: test
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gb s3://test
upload: ./file_1gb to s3://test/file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-13 15:24:38 1048576000 file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]#

[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-1589 /]# hctl status
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 1286888725
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-rhev4-2451
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1588
    [started]  hax                 0x7200000000000001:0x36         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1588@22001
    [started]  rgw_s3              0x7200000000000001:0x37         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1588@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  rgw_s3              0x7200000000000001:0x39         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x3a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  rgw_s3              0x7200000000000001:0x3b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x3c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  rgw_s3              0x7200000000000001:0x3d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x3e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  rgw_s3              0x7200000000000001:0x3f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1587
    [started]  hax                 0x7200000000000001:0x40         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1587@22001
    [started]  rgw_s3              0x7200000000000001:0x41         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1587@21001
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1587
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1588
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22002
    cortx-data-headless-svc-ssc-vm-rhev4-2450
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1589  (RC)
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@22001
    [started]  ioservice           0x7200000000000001:0x29         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@21002
    [started]  confd               0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  rgw_s3              0x7200000000000001:0x2d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2450
    [started]  hax                 0x7200000000000001:0x2e         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2450@22001
    [started]  rgw_s3              0x7200000000000001:0x2f         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2450@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2451
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2451@22001
    [started]  rgw_s3              0x7200000000000001:0x31         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2451@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x32         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  rgw_s3              0x7200000000000001:0x33         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1589
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1589@22001
    [started]  rgw_s3              0x7200000000000001:0x35         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1589@21001
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-1589 /]#

[root@ssc-vm-g4-rhev4-0710 ~]# kubectl get deployment cortx-data-ssc-vm-g4-rhev4-0712 -o yaml > cortx-data-ssc-vm-g4-rhev4-0712.yaml
[root@ssc-vm-g4-rhev4-0710 ~]# kubectl delete deployment cortx-data-ssc-vm-g4-rhev4-0712
deployment.apps "cortx-data-ssc-vm-g4-rhev4-0712" deleted
[root@ssc-vm-g4-rhev4-0710 ~]#

[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-1589 /]# hctl status -d | grep offline
    [offline]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [offline]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [offline]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [offline]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    [offline]  /dev/sdd
    [offline]  /dev/sde
    [offline]  /dev/sdc
    [offline]  /dev/sdg
    [offline]  /dev/sdh
    [offline]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-1589 /]#

[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gbDn s3://test
upload: ./file_1gbDn to s3://test/file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-13 15:24:38 1048576000 file_1gb
2022-07-13 15:33:52 1048576000 file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]#
[root@ssc-vm-g4-rhev4-0710 ~]# kubectl apply -f cortx-data-ssc-vm-g4-rhev4-0712.yaml
deployment.apps/cortx-data-ssc-vm-g4-rhev4-0712 created
[root@ssc-vm-g4-rhev4-0710 ~]#
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-1589 /]# hctl status -d | egrep 'offline|dtm_recovering|UNKNOWN'
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-1589 /]#

[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls
2022-07-13 15:24:14 test
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-13 15:24:38 1048576000 file_1gb
2022-07-13 15:33:52 1048576000 file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]#
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gbDn1 s3://test
upload: ./file_1gbDn1 to s3://test/file_1gbDn1
[root@ssc-vm-g4-rhev4-0710 ~]#
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-13 15:24:38 1048576000 file_1gb
2022-07-13 15:33:52 1048576000 file_1gbDn
2022-07-13 15:39:09 1048576000 file_1gbDn1
[root@ssc-vm-g4-rhev4-0710 ~]#

Node restart

[root@ssc-vm-g4-rhev4-0710 ~]# kubectl delete pod cortx-data-ssc-vm-g4-rhev4-0711-bd475fcb5-p9qr8
pod "cortx-data-ssc-vm-g4-rhev4-0711-bd475fcb5-p9qr8" deleted
[root@ssc-vm-g4-rhev4-0710 ~]#

[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-13 15:24:38 1048576000 file_1gb
2022-07-13 15:33:52 1048576000 file_1gbDn
2022-07-13 15:39:09 1048576000 file_1gbDn1
[root@ssc-vm-g4-rhev4-0710 ~]#
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gb s3://test/file_1gb_restart
upload: ./file_1gb to s3://test/file_1gb_restart
[root@ssc-vm-g4-rhev4-0710 ~]#
mssawant commented 2 years ago

retest this please.

supriyachavan4398 commented 2 years ago

With custom build https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7214/ Did 27N deployment with configuration SNS: 18+9+0, DIX: 1+26+0 at https://eos-jenkins.colo.seagate.com/job/Cortx-kubernetes/job/setup-cortx-cluster-solution-input/615/ with increased consul resource limits:

      consul:
        server:
          storage: 10Gi
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 2Gi
              cpu: 800m

Degraded IOs: Node failure and degraded write test

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS    RESTARTS      AGE
cortx-consul-client-4v4xz                            1/1     Running   0             73m
cortx-consul-client-4vcwt                            1/1     Running   0             73m
cortx-consul-client-5kqsv                            1/1     Running   0             73m
cortx-consul-client-5z7qs                            1/1     Running   0             73m
cortx-consul-client-65z2g                            1/1     Running   0             73m
cortx-consul-client-74xpm                            1/1     Running   0             73m
cortx-consul-client-9lvqr                            1/1     Running   0             73m
cortx-consul-client-9s8q4                            1/1     Running   0             73m
cortx-consul-client-9ttx5                            1/1     Running   0             73m
cortx-consul-client-bbtpd                            1/1     Running   0             73m
cortx-consul-client-dcvrx                            1/1     Running   0             74m
cortx-consul-client-g58dt                            1/1     Running   0             73m
cortx-consul-client-gjml4                            1/1     Running   0             73m
cortx-consul-client-h46ds                            1/1     Running   0             74m
cortx-consul-client-js5hp                            1/1     Running   0             73m
cortx-consul-client-ks5nw                            1/1     Running   0             73m
cortx-consul-client-kvv9s                            1/1     Running   0             73m
cortx-consul-client-l88ss                            1/1     Running   0             73m
cortx-consul-client-p5b2d                            1/1     Running   0             73m
cortx-consul-client-phk29                            1/1     Running   0             73m
cortx-consul-client-rqgq5                            1/1     Running   0             73m
cortx-consul-client-v7x2l                            1/1     Running   0             73m
cortx-consul-client-vcf62                            1/1     Running   0             73m
cortx-consul-client-vg2ll                            1/1     Running   0             73m
cortx-consul-client-xdn4j                            1/1     Running   0             74m
cortx-consul-client-xh86n                            1/1     Running   0             74m
cortx-consul-client-zhm27                            1/1     Running   0             73m
cortx-consul-server-0                                1/1     Running   0             73m
cortx-consul-server-1                                1/1     Running   0             74m
cortx-consul-server-2                                1/1     Running   0             74m
cortx-control-776bdb7c78-n77ms                       1/1     Running   0             72m
cortx-data-ssc-vm-g2-rhev4-1631-6585778847-tmdh4     4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-1632-846b5b978-8kzqs      4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-1635-96d59fc5f-hxxbv      4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-2237-869f5db498-d5b5l     4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-2238-54f6d48695-p9zrd     4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3031-564c4f7f-hkdjb       4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3156-86f4c7ddff-nsxvs     4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3157-c47bbc7b4-6kjbq      4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3158-bd56bd8bf-hjp2n      4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3159-565ccb6946-cckb5     4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3160-9bd88f5b-llnlj       4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3166-6fbc5558b-spxp8      4/4     Running   0             71m
cortx-data-ssc-vm-g2-rhev4-3170-f5f7578d5-xst29      4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2107-f7f467fcf-9t7fk      4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2184-744674f9cd-pl925     4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2198-7745f7d9f-hzsmb      4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2278-75689f7976-w92xb     4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2279-958d75cbc-nf667      4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2281-7dfcb868b8-rmjs6     4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2282-7855c56965-2p7p7     4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2283-7b749ddbd6-lbzxt     4/4     Running   0             71m
cortx-data-ssc-vm-g3-rhev4-2284-599dbbc999-84dw6     4/4     Running   0             71m
cortx-data-ssc-vm-g4-rhev4-1583-8757ff9f-c6lr6       4/4     Running   0             71m
cortx-data-ssc-vm-g4-rhev4-1590-864b9898dc-6q66b     4/4     Running   0             71m
cortx-data-ssc-vm-g4-rhev4-1591-64cdf9674d-fc7wp     4/4     Running   0             71m
cortx-data-ssc-vm-g4-rhev4-1592-578c4cd585-2wm4h     4/4     Running   0             71m
cortx-data-ssc-vm-rhev4-2635-65584ff8d8-k79cv        4/4     Running   0             71m
cortx-ha-8496df8565-tb89q                            3/3     Running   0             59m
cortx-kafka-0                                        1/1     Running   1 (75m ago)   76m
cortx-kafka-1                                        1/1     Running   0             76m
cortx-kafka-2                                        1/1     Running   0             76m
cortx-server-ssc-vm-g2-rhev4-1631-7f7759bb6b-524kr   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-1632-7b4748948d-p4slr   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-1635-7bd9f9f9bb-h4gtr   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-2237-8454ff755b-2mdpz   2/2     Running   5 (32m ago)   67m
cortx-server-ssc-vm-g2-rhev4-2238-5fcbf5d7bb-chfs7   2/2     Running   5 (32m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3031-86fb7866-fg9gx     2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3156-6945b74f78-mpbl2   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3157-b88b8b5bd-ns8np    2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3158-6ffb75b58-j2gsh    2/2     Running   5 (32m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3159-58f698f85b-mb5jt   2/2     Running   4 (38m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3160-6c7c64676-vcqlz    2/2     Running   5 (32m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3166-76bb4f8ffb-84t2z   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g2-rhev4-3170-799cdf96f8-lglvw   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2107-5d5fbfb866-tr7x2   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2184-58cbd7744d-n9hs9   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2198-67cb6c555d-x4ccl   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2278-5c96498b6d-n9v75   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2279-78975584db-zv85d   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2281-7ccbb8bb7b-m2kq7   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2282-6c5589b6d6-qbb92   2/2     Running   5 (32m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2283-5c84b5c7b4-79nh6   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g3-rhev4-2284-7d44d85b46-zg2gv   2/2     Running   5 (32m ago)   67m
cortx-server-ssc-vm-g4-rhev4-1583-54789b7f9d-lwgmj   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g4-rhev4-1590-85fb58cc98-9f5ff   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g4-rhev4-1591-7b49b78df6-kbkkz   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-g4-rhev4-1592-5cdf8684fd-msfwd   2/2     Running   6 (26m ago)   67m
cortx-server-ssc-vm-rhev4-2635-6bd6cb79bd-58sgq      2/2     Running   6 (26m ago)   67m
cortx-zookeeper-0                                    1/1     Running   0             76m
cortx-zookeeper-1                                    1/1     Running   0             76m
cortx-zookeeper-2                                    1/1     Running   0             76m

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1631-6585778847-tmdh4 -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1592
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x40         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  ioservice           0x7200000000000001:0x41         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21001
    [started]  ioservice           0x7200000000000001:0x42         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21002
    [started]  confd               0x7200000000000001:0x43         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0x44         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  ioservice           0x7200000000000001:0x45         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21001
    [started]  ioservice           0x7200000000000001:0x46         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21002
    [started]  confd               0x7200000000000001:0x47         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  ioservice           0x7200000000000001:0x49         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21001
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21002
    [started]  confd               0x7200000000000001:0x4b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3159
    [started]  hax                 0x7200000000000001:0x4c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@22001
    [started]  ioservice           0x7200000000000001:0x4d         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21001
    [started]  ioservice           0x7200000000000001:0x4e         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@21002
    [started]  confd               0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3159@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x50         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  ioservice           0x7200000000000001:0x51         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21002
    [started]  confd               0x7200000000000001:0x53         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3166
    [started]  hax                 0x7200000000000001:0x54         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3166@22001
    [started]  ioservice           0x7200000000000001:0x55         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3166@21001
    [started]  ioservice           0x7200000000000001:0x56         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3166@21002
    [started]  confd               0x7200000000000001:0x57         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3166@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x58         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  ioservice           0x7200000000000001:0x59         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21001
    [started]  ioservice           0x7200000000000001:0x5a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21002
    [started]  confd               0x7200000000000001:0x5b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22002
    cortx-data-headless-svc-ssc-vm-rhev4-2635
    [started]  hax                 0x7200000000000001:0x5c         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2635@22001
    [started]  ioservice           0x7200000000000001:0x5d         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2635@21001
    [started]  ioservice           0x7200000000000001:0x5e         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2635@21002
    [started]  confd               0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2635@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x60         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x61         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0x62         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0x63         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x64         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0x65         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0x66         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0x67         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x68         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x6a         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x6b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22002
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    [started]  hax                 0x7200000000000001:0x6c         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  rgw_s3              0x7200000000000001:0x6d         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1583
    [started]  hax                 0x7200000000000001:0x6e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1583@22001
    [started]  rgw_s3              0x7200000000000001:0x6f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1583@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3159
    [started]  hax                 0x7200000000000001:0x70         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3159@22001
    [started]  rgw_s3              0x7200000000000001:0x71         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3159@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x72         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  rgw_s3              0x7200000000000001:0x73         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2635
    [started]  hax                 0x7200000000000001:0x74         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2635@22001
    [started]  rgw_s3              0x7200000000000001:0x75         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2635@21001
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22002
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x76         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  rgw_s3              0x7200000000000001:0x77         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3166
    [started]  hax                 0x7200000000000001:0x78         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3166@22001
    [started]  rgw_s3              0x7200000000000001:0x79         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3166@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1592
    [started]  hax                 0x7200000000000001:0x7a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1592@22001
    [started]  rgw_s3              0x7200000000000001:0x7b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1592@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x7c         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw_s3              0x7200000000000001:0x7d         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x7e         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw_s3              0x7200000000000001:0x7f         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x80         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw_s3              0x7200000000000001:0x81         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0x82         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  rgw_s3              0x7200000000000001:0x83         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x84         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw_s3              0x7200000000000001:0x85         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x86         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  rgw_s3              0x7200000000000001:0x87         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x88         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw_s3              0x7200000000000001:0x89         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x8a         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw_s3              0x7200000000000001:0x8b         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x8c         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw_s3              0x7200000000000001:0x8d         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x8e         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw_s3              0x7200000000000001:0x8f         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x90         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  rgw_s3              0x7200000000000001:0x91         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x92         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw_s3              0x7200000000000001:0x93         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x94         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  rgw_s3              0x7200000000000001:0x95         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21001
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22002
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x96         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw_s3              0x7200000000000001:0x97         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x98         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw_s3              0x7200000000000001:0x99         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x9a         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw_s3              0x7200000000000001:0x9b         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1591
    [started]  hax                 0x7200000000000001:0x9c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1591@22001
    [started]  rgw_s3              0x7200000000000001:0x9d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1591@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x9e         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw_s3              0x7200000000000001:0x9f         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1590
    [started]  hax                 0x7200000000000001:0xa0         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1590@22001
    [started]  rgw_s3              0x7200000000000001:0xa1         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1590@21001
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031  (RC)
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1583
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1590
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@22001
    [started]  ioservice           0x7200000000000001:0x29         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@21002
    [started]  confd               0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x2d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x2f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x31         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x32         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x33         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1591
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@22001
    [started]  ioservice           0x7200000000000001:0x35         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@21001
    [started]  ioservice           0x7200000000000001:0x36         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@21002
    [started]  confd               0x7200000000000001:0x37         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x39         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x3a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x3c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x3d         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0x3e         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0x3f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22002
Devices:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1592
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3159
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3166
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-rhev4-2635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1583
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3159
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    cortx-server-headless-svc-ssc-vm-rhev4-2635
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3166
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1592
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1591
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1590
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1583
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1590
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1591
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 mb s3://test --endpoint-url http://$IP:$Port
make_bucket: test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-14 02:08:21 test
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Write file..."
Write file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-14 02:08:54 1048576000 file_1gb

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 0,
    "healthy": 1051372201
  }
}

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get deployment cortx-data-ssc-vm-g2-rhev4-1632 -o yaml > cortx-data-ssc-vm-g2-rhev4-1632.yaml
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl delete deployment cortx-data-ssc-vm-g2-rhev4-1632
deployment.apps "cortx-data-ssc-vm-g2-rhev4-1632" deleted

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1631-6585778847-tmdh4 -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d | grep -i "offline"
    [offline]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [offline]  ioservice           0x7200000000000001:0x39         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [offline]  ioservice           0x7200000000000001:0x3a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [offline]  confd               0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22002
    [offline]  /dev/sdd
    [offline]  /dev/sde
    [offline]  /dev/sdc
    [offline]  /dev/sdg
    [offline]  /dev/sdh
    [offline]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 1051372201,
    "healthy": 0
  }
}

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-14 02:08:54 1048576000 file_1gb
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-14 02:08:21 test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb1 --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb1
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-14 02:08:54 1048576000 file_1gb
2022-07-14 02:21:08 1048576000 file_1gb1
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Read file..."
Read file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp s3://test/file_1gb file_1gbDn --endpoint-url http://$IP:$Port
download: s3://test/file_1gb to ./file_1gbDn
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp s3://test/file_1gb1 file_1gbDn1 --endpoint-url http://$IP:$Port
download: s3://test/file_1gb1 to ./file_1gbDn1
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl apply -f cortx-data-ssc-vm-g2-rhev4-1632.yaml
deployment.apps/cortx-data-ssc-vm-g2-rhev4-1632 created

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port

Read timeout on endpoint URL: "http://10.100.173.31:80/"
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port

Read timeout on endpoint URL: "http://10.100.173.31:80/"

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-3031 2a0aea9a7ac8f1fd8c121260871a709a]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 2101346298,
    "healthy": 0
  }
}

After the node restarts, It was failing to update byte counts and found an HAconsistency exception for the byte-count updater thread in hare-hax logs of RC node. Also, It was not able to list down the bucket list. 

2022-07-14 08:08:04,447 [DEBUG] {byte-count-updater} KVGET key=ioservices/, kwargs={'recurse': True}
2022-07-14 08:08:04,454 [ERROR] {byte-count-updater} Failed to update Consul KV due to an intermittent error. The error is swallowed since new attempts will be made timely
Traceback (most recent call last):
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/bytecount.py", line 150, in _execute
    pver_items = self._get_pver_with_pver_status(motr)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/bytecount.py", line 65, in _get_pver_with_pver_status
    iosservice_items = self.consul.kv.kv_get('ioservices/', recurse=True)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/consul/cache.py", line 128, in wrapper
    ret_value = f(*args, **kwds)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/util.py", line 224, in kv_get
    raise HAConsistencyException('Could not get data from Consul KV')
hax.exception.HAConsistencyException

motr[00100]:  dc40  ERROR  [spiel/cmd.c:2126:spiel_proc_counter_item_rlink_cb]  connect failed
2022-07-14 09:29:51,225 [ERROR] {byte-count-updater} Failed due to Bytecount stats unavailable. Aborting this iteration. Waiting for next attempt.
Traceback (most recent call last):
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/bytecount.py", line 144, in _execute
    motr.get_proc_bytecount(ios)
  File "/opt/seagate/cortx/hare/lib64/python3.6/site-packages/hax/motr/__init__.py", line 761, in get_proc_bytecount
    raise BytecountException('Bytecount stats unavailable')
hax.exception.BytecountException
motr[00100]:  bad0   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=32 hl=0x7f3bcc01e7f0 ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 lq_tags=(confirmed=775 delivered=775 next=775 assign=779)
motr[00100]:  bad0   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=32 hl=0x7f3bb401d740 ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21001 lq_tags=(confirmed=775 delivered=775 next=775 assign=779)
motr[00100]:  bad0   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=32 hl=0x7f3b90010810 ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22002 lq_tags=(confirmed=769 delivered=769 next=769 assign=773)
motr[00100]:  5a10  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f3bcc01ff88 dest_ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 ri_session=0x7f3bcc01ed28 ri_nr_sent_max=10 ri_deadline=1657791130571285858 ri_nr_sent=2
motr[00100]:  5a10  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f3b8c009760, item 0x7f3bcc01ff88[207] failed with ri_error=-110
motr[00100]:  5a10  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f3bcc01ff88 dest_ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 ri_session=0x7f3bcc01ed28 ri_nr_sent_max=10 ri_deadline=1657791140997259498 ri_nr_sent=1
motr[00100]:  5a10  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f3c68086990, item 0x7f3bcc01ff88[207] failed with ri_error=-110
motr[00100]:  5a10  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f3bcc01ff88 dest_ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 ri_session=0x7f3bcc01ed28 ri_nr_sent_max=10 ri_deadline=1657791150672737079 ri_nr_sent=1
motr[00100]:  5a10  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f3c68086990, item 0x7f3bcc01ff88[207] failed with ri_error=-110
motr[00100]:  5a10  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f3bcc01ff88 dest_ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 ri_session=0x7f3bcc01ed28 ri_nr_sent_max=10 ri_deadline=1657791160297477124 ri_nr_sent=1
motr[00100]:  5a10  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f3c6805dc90, item 0x7f3bcc01ff88[207] failed with ri_error=-110
motr[00100]:  5a10  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f3bcc01ff88 dest_ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 ri_session=0x7f3bcc01ed28 ri_nr_sent_max=10 ri_deadline=1657791170091416207 ri_nr_sent=1
motr[00100]:  5a10  ERROR  [rpc/frmops.c:486:item_done]  packet 0x7f3c6805dc90, item 0x7f3bcc01ff88[207] failed with ri_error=-110
motr[00100]:  5a10  ERROR  [rpc/frmops.c:482:item_done]   item: 0x7f3bcc01ff88 dest_ep=inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002 ri_session=0x7f3bcc01ed28 ri_nr_sent_max=10 ri_deadline=1657791179173119271 ri_nr_sent=1

cc. @mssawant, @vaibhavparatwar

mssawant commented 2 years ago

Seeing a Motr communication issue when node comes back online. Working with motr team, cc @yeshpal-jain-seagate @madhavemuri

mssawant commented 2 years ago

15N deployment and testing

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS     RESTARTS      AGE
cortx-consul-client-4p8h7                            1/1     Running    0             15m
cortx-consul-client-65p5w                            1/1     Running    0             15m
cortx-consul-client-8rl92                            1/1     Running    0             14m
cortx-consul-client-b9jml                            1/1     Running    0             14m
cortx-consul-client-f24kf                            1/1     Running    0             14m
cortx-consul-client-fbvnp                            1/1     Running    0             14m
cortx-consul-client-ggjfg                            1/1     Running    0             14m
cortx-consul-client-hvvxb                            1/1     Running    0             14m
cortx-consul-client-jl2wr                            1/1     Running    0             14m
cortx-consul-client-lr72p                            1/1     Running    0             14m
cortx-consul-client-m9h2n                            1/1     Running    0             14m
cortx-consul-client-p4trh                            1/1     Running    0             14m
cortx-consul-client-pdrrs                            1/1     Running    0             14m
cortx-consul-client-q58f9                            1/1     Running    0             14m
cortx-consul-client-x5zzc                            1/1     Running    0             15m
cortx-consul-server-0                                1/1     Running    0             14m
cortx-consul-server-1                                1/1     Running    0             14m
cortx-consul-server-2                                1/1     Running    0             15m
cortx-control-7dff4fb58b-pmt8d                       1/1     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-1630-6d57c48668-xm88c     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-1631-959d697b7-wh7c6      4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-1632-7df55b8498-9tqg4     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-1635-6d5fddd855-j88rd     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-2237-798bb76cf8-x5d4v     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-2238-579dd7d995-vpf4m     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-3031-6c6d948d79-64m6w     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-3156-8458bcc9c9-czwrs     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-3157-846cdc7ffb-jbkfp     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-3158-76d9f686d5-5cszv     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-3160-744687568d-t9vq2     4/4     Running    0             12m
cortx-data-ssc-vm-g2-rhev4-3170-74fb88f8f-j56sw      4/4     Running    0             12m
cortx-data-ssc-vm-g3-rhev4-2107-78c4d7db7f-sctts     4/4     Running    0             11m
cortx-data-ssc-vm-g3-rhev4-2184-7588f5b466-vlgp6     4/4     Running    0             11m
cortx-data-ssc-vm-g3-rhev4-2198-69d9ddbbf5-5txhz     4/4     Running    0             11m
cortx-ha-d6454c9b7-6x2vx                             0/3     Init:0/1   0             4m55s
cortx-kafka-0                                        1/1     Running    1 (16m ago)   17m
cortx-kafka-1                                        1/1     Running    0             17m
cortx-kafka-2                                        1/1     Running    0             17m
cortx-server-ssc-vm-g2-rhev4-1630-5dc8d4c9f7-27kvx   2/2     Running    0             9m9s
cortx-server-ssc-vm-g2-rhev4-1631-b859cfff5-zdfj5    2/2     Running    0             9m9s
cortx-server-ssc-vm-g2-rhev4-1632-5cc475ddc7-phktq   2/2     Running    0             9m8s
cortx-server-ssc-vm-g2-rhev4-1635-95d89b56c-92tlp    2/2     Running    0             9m7s
cortx-server-ssc-vm-g2-rhev4-2237-7ff5d79dbf-cxsv4   2/2     Running    0             9m6s
cortx-server-ssc-vm-g2-rhev4-2238-577b7d8bf9-ftkjm   2/2     Running    0             9m5s
cortx-server-ssc-vm-g2-rhev4-3031-df984777f-kf9wk    2/2     Running    0             9m4s
cortx-server-ssc-vm-g2-rhev4-3156-6498d7ccc9-f2hlq   2/2     Running    0             9m3s
cortx-server-ssc-vm-g2-rhev4-3157-559dd5b8d5-kzft8   2/2     Running    0             9m2s
cortx-server-ssc-vm-g2-rhev4-3158-79dc7dd767-w4d9l   2/2     Running    0             9m2s
cortx-server-ssc-vm-g2-rhev4-3160-69946dbf5c-jj4qv   2/2     Running    0             9m1s
cortx-server-ssc-vm-g2-rhev4-3170-55b9ccc899-22nq7   2/2     Running    0             9m
cortx-server-ssc-vm-g3-rhev4-2107-548d48b687-82k85   2/2     Running    0             8m59s
cortx-server-ssc-vm-g3-rhev4-2184-6dffb7df5c-8wrjb   2/2     Running    0             8m58s
cortx-server-ssc-vm-g3-rhev4-2198-b9b96995-sjbrg     2/2     Running    0             8m58s
cortx-zookeeper-0                                    1/1     Running    0             17m
cortx-zookeeper-1                                    1/1     Running    0             17m
cortx-zookeeper-2                                    1/1     Running    0             17m
[root@ssc-vm-g2-rhev4-1630 ~]#

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1630-6d57c48668-xm88c -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22002
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    [started]  hax                 0x7200000000000001:0x3e         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  rgw_s3              0x7200000000000001:0x3f         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x40         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  rgw_s3              0x7200000000000001:0x41         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x42         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw_s3              0x7200000000000001:0x43         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x44         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw_s3              0x7200000000000001:0x45         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x46         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  rgw_s3              0x7200000000000001:0x47         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x48         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  rgw_s3              0x7200000000000001:0x49         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x4a         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw_s3              0x7200000000000001:0x4b         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x4c         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw_s3              0x7200000000000001:0x4d         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x4e         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  rgw_s3              0x7200000000000001:0x4f         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x50         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw_s3              0x7200000000000001:0x51         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0x52         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  rgw_s3              0x7200000000000001:0x53         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x54         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw_s3              0x7200000000000001:0x55         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x56         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw_s3              0x7200000000000001:0x57         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1630
    [started]  hax                 0x7200000000000001:0x58         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1630@22001
    [started]  rgw_s3              0x7200000000000001:0x59         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1630@21001
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031  (RC)
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1630
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  ioservice           0x7200000000000001:0x29         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21002
    [started]  confd               0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x2d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x2f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x31         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0x32         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0x33         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0x35         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0x36         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0x37         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  ioservice           0x7200000000000001:0x39         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21001
    [started]  ioservice           0x7200000000000001:0x3a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21002
    [started]  confd               0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22002
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x3c         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw_s3              0x7200000000000001:0x3d         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21001
Devices:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1630
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1630
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 mb s3://test
make_bucket: test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp 1G s3://test
upload: ./1G to s3://test/1G
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test
2022-07-15 12:13:46 1073741824 1G
[root@ssc-vm-g2-rhev4-1630 ~]#

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get deployment cortx-data-ssc-vm-g2-rhev4-1635 -o yaml > cortx-data-ssc-vm-g2-rhev4-1635.yaml
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl delete deployment cortx-data-ssc-vm-g2-rhev4-1635
deployment.apps "cortx-data-ssc-vm-g2-rhev4-1635" deleted
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls
2022-07-15 12:13:23 test
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test
2022-07-15 12:13:46 1073741824 1G
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp 1G s3://test/1G_node_DN
upload: ./1G to s3://test/1G_node_DN
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1630-6d57c48668-xm88c -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 1073741824
    healthy : 1073741824
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d | egrep 'offline|dtm_recovering|UNKNOWN'
    [offline]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [offline]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [offline]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [offline]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22002
    [offline]  /dev/sdd
    [offline]  /dev/sde
    [offline]  /dev/sdc
    [offline]  /dev/sdg
    [offline]  /dev/sdh
    [offline]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]#
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl apply -f cortx-data-ssc-vm-g2-rhev4-1635.yaml
deployment.apps/cortx-data-ssc-vm-g2-rhev4-1635 created
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d | egrep 'offline|dtm_recovering|UNKNOWN'
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls
2022-07-15 12:13:23 test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test
2022-07-15 12:13:46 1073741824 1G
2022-07-15 12:56:57 1073741824 1G_node_DN
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp 1G s3://test/1G_node_fail_UP
upload: ./1G to s3://test/1G_node_fail_UP
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test
2022-07-15 12:13:46 1073741824 1G
2022-07-15 12:56:57 1073741824 1G_node_DN
2022-07-15 13:08:14 1073741824 1G_node_fail_UP
[root@ssc-vm-g2-rhev4-1630 ~]#

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 3221225472
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl delete pod cortx-data-ssc-vm-g2-rhev4-1632-7df55b8498-9tqg4
pod "cortx-data-ssc-vm-g2-rhev4-1632-7df55b8498-9tqg4" deleted
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls
2022-07-15 12:13:23 test
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1630-6d57c48668-xm88c -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d | egrep 'offline|dtm_recovering|UNKNOWN'
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]#
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 3221225472
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# exit
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test
2022-07-15 12:13:46 1073741824 1G
2022-07-15 12:56:57 1073741824 1G_node_DN
2022-07-15 13:08:14 1073741824 1G_node_fail_UP
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp 1G s3://test/1G_node_restart
upload: ./1G to s3://test/1G_node_restart
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test
2022-07-15 12:13:46 1073741824 1G
2022-07-15 12:56:57 1073741824 1G_node_DN
2022-07-15 13:08:14 1073741824 1G_node_fail_UP
2022-07-15 13:30:05 1073741824 1G_node_restart
[root@ssc-vm-g2-rhev4-1630 ~]#
mssawant commented 2 years ago

retest this please.

mssawant commented 2 years ago

restest this please.

mssawant commented 2 years ago

6N and 15N deployment, regular IO, degraded IO with node restart and failure worked fine with build https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7255

pavankrishnat commented 2 years ago

retest this please

supriyachavan4398 commented 2 years ago

With custom build https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7261/ Did 25N deployment with configuration SNS: 18+9+0, DIX: 1+22+0 at https://eos-jenkins.colo.seagate.com/job/Cortx-kubernetes/job/setup-cortx-cluster-solution-input/631/ with increased consul resource limits:

      consul:
        server:
          storage: 10Gi
          resources:
            requests:
              memory: 1Gi
              cpu: 500m
            limits:
              memory: 2Gi
              cpu: 800m
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS    RESTARTS       AGE
cortx-consul-client-29xdc                            1/1     Running   0              125m
cortx-consul-client-2rl4t                            1/1     Running   0              125m
cortx-consul-client-5qqsv                            1/1     Running   0              125m
cortx-consul-client-8sqqh                            1/1     Running   0              125m
cortx-consul-client-9fcvw                            1/1     Running   0              125m
cortx-consul-client-9mlqz                            1/1     Running   0              125m
cortx-consul-client-cnr2g                            1/1     Running   0              125m
cortx-consul-client-csmp8                            1/1     Running   0              126m
cortx-consul-client-hzgzq                            1/1     Running   0              126m
cortx-consul-client-jd9g4                            1/1     Running   0              125m
cortx-consul-client-kfk25                            1/1     Running   0              125m
cortx-consul-client-m2s5h                            1/1     Running   0              126m
cortx-consul-client-nk9nv                            1/1     Running   0              126m
cortx-consul-client-qtnjx                            1/1     Running   0              125m
cortx-consul-client-r92tf                            1/1     Running   0              125m
cortx-consul-client-sk59v                            1/1     Running   0              125m
cortx-consul-client-svq4t                            1/1     Running   0              125m
cortx-consul-client-sxz5k                            1/1     Running   0              125m
cortx-consul-client-tssq5                            1/1     Running   0              125m
cortx-consul-client-vdz9b                            1/1     Running   0              125m
cortx-consul-client-vlhg7                            1/1     Running   0              125m
cortx-consul-client-vmghb                            1/1     Running   0              126m
cortx-consul-client-vmm86                            1/1     Running   0              127m
cortx-consul-client-z6z5z                            1/1     Running   0              125m
cortx-consul-server-0                                1/1     Running   0              125m
cortx-consul-server-1                                1/1     Running   0              126m
cortx-consul-server-2                                1/1     Running   0              127m
cortx-control-86f648f6f6-6rfbv                       1/1     Running   0              124m
cortx-data-ssc-vm-g2-rhev4-1631-648c5684c5-nmwzw     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-1632-956ddb57d-9rk4s      4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-1635-6dd69f4fdc-z94lz     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-2237-79cb97cdbb-7jfzd     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-2238-6d6895dd7f-6v6dd     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-3031-5f4f74f959-bq9rq     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-3156-74f9f6cb6c-5wh2m     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-3157-765dcf555b-wvmrz     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-3158-67cc5f64b5-mgt2c     4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-3160-5776c8ff4-qn7bh      4/4     Running   0              123m
cortx-data-ssc-vm-g2-rhev4-3170-5dc44c6865-dlkf4     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2107-5dd8d8646c-csc2s     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2184-7b4f45ff9d-x75bs     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2198-64cff9fb75-2xsg7     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2278-c684ddbb8-dd6dp      4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2279-76d6dd6f5-sxp6m      4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2281-79c98d9748-29gt6     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2282-7984cc8cd9-rbj4z     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2283-85c6cbf5d6-wkdsv     4/4     Running   0              123m
cortx-data-ssc-vm-g3-rhev4-2284-95df4b4f9-dlwf6      4/4     Running   0              123m
cortx-data-ssc-vm-g4-rhev4-1583-58d7854b99-7vzpx     4/4     Running   0              123m
cortx-data-ssc-vm-g4-rhev4-1590-678744cfc5-2lvss     4/4     Running   0              123m
cortx-data-ssc-vm-g4-rhev4-1591-6574485f68-bndcz     4/4     Running   0              123m
cortx-data-ssc-vm-g4-rhev4-1592-79d7fb487c-tpctq     4/4     Running   0              123m
cortx-ha-57c75dc58f-zbjkj                            3/3     Running   0              115m
cortx-kafka-0                                        1/1     Running   2 (128m ago)   128m
cortx-kafka-1                                        1/1     Running   2 (127m ago)   128m
cortx-kafka-2                                        1/1     Running   2 (127m ago)   128m
cortx-server-ssc-vm-g2-rhev4-1631-986f6fc78-f2bbq    2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-1632-86d657c856-psp48   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-1635-674fbb79f8-dr8l5   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-2237-c59df4c6-7l954     2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-2238-795b6f8958-9qbts   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-3031-676c9446c8-47sk9   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-3156-c9bcf8746-hcfsg    2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-3157-7c87c5dfbd-d8vm9   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-3158-5d6ddfc866-c8kfs   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-3160-84459c7cd8-k8bvx   2/2     Running   0              120m
cortx-server-ssc-vm-g2-rhev4-3170-54c8bcb59d-l25jw   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2107-5956c77bb6-gkb2c   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2184-6f774975d4-qtfz6   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2198-654dcbfc4-fgrds    2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2278-64977755d8-4c229   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2279-789874df46-9nkmw   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2281-85bdb585c6-jkbfv   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2282-8689f7c57d-4rpwv   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2283-578cb8bcc6-c8rlv   2/2     Running   0              120m
cortx-server-ssc-vm-g3-rhev4-2284-6d4dc95498-vx8tv   2/2     Running   0              120m
cortx-server-ssc-vm-g4-rhev4-1583-656499c65d-s65m8   2/2     Running   0              120m
cortx-server-ssc-vm-g4-rhev4-1590-578878b5c4-x4g9p   2/2     Running   0              120m
cortx-server-ssc-vm-g4-rhev4-1591-5dcbcfd66d-xdrch   2/2     Running   0              120m
cortx-server-ssc-vm-g4-rhev4-1592-7468b8c94b-7q7wk   2/2     Running   0              119m
cortx-zookeeper-0                                    1/1     Running   0              128m
cortx-zookeeper-1                                    1/1     Running   0              128m
cortx-zookeeper-2                                    1/1     Running   0              128m
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1631-648c5684c5-nmwzw -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x40         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x41         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x42         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x43         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x44         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  ioservice           0x7200000000000001:0x45         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21001
    [started]  ioservice           0x7200000000000001:0x46         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21002
    [started]  confd               0x7200000000000001:0x47         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x49         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x4b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x4c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x4d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0x4e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x50         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x51         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0x53         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1583
    [started]  hax                 0x7200000000000001:0x54         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@22001
    [started]  ioservice           0x7200000000000001:0x55         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@21001
    [started]  ioservice           0x7200000000000001:0x56         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@21002
    [started]  confd               0x7200000000000001:0x57         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x58         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0x59         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x5a         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x5b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x5c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x5d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x5e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22002
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x60         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw_s3              0x7200000000000001:0x61         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1591
    [started]  hax                 0x7200000000000001:0x62         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1591@22001
    [started]  rgw_s3              0x7200000000000001:0x63         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1591@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1583
    [started]  hax                 0x7200000000000001:0x64         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1583@22001
    [started]  rgw_s3              0x7200000000000001:0x65         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1583@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x66         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw_s3              0x7200000000000001:0x67         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x68         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw_s3              0x7200000000000001:0x69         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    [started]  hax                 0x7200000000000001:0x6a         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  rgw_s3              0x7200000000000001:0x6b         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x6c         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw_s3              0x7200000000000001:0x6d         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1590
    [started]  hax                 0x7200000000000001:0x6e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1590@22001
    [started]  rgw_s3              0x7200000000000001:0x6f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1590@21001
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22002
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x70         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw_s3              0x7200000000000001:0x71         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x72         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  rgw_s3              0x7200000000000001:0x73         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x74         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw_s3              0x7200000000000001:0x75         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x76         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw_s3              0x7200000000000001:0x77         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0x78         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  rgw_s3              0x7200000000000001:0x79         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x7a         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw_s3              0x7200000000000001:0x7b         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x7c         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  rgw_s3              0x7200000000000001:0x7d         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1592
    [started]  hax                 0x7200000000000001:0x7e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1592@22001
    [started]  rgw_s3              0x7200000000000001:0x7f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1592@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x80         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw_s3              0x7200000000000001:0x81         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x82         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  rgw_s3              0x7200000000000001:0x83         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x84         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  rgw_s3              0x7200000000000001:0x85         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x86         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw_s3              0x7200000000000001:0x87         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x88         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  rgw_s3              0x7200000000000001:0x89         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x8a         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw_s3              0x7200000000000001:0x8b         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x8c         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw_s3              0x7200000000000001:0x8d         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x8e         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw_s3              0x7200000000000001:0x8f         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21001
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283  (RC)
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1590
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  ioservice           0x7200000000000001:0x29         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21002
    [started]  confd               0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x2d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x2f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1592
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@22001
    [started]  ioservice           0x7200000000000001:0x31         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@21001
    [started]  ioservice           0x7200000000000001:0x32         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@21002
    [started]  confd               0x7200000000000001:0x33         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0x35         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0x36         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0x37         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1591
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@22001
    [started]  ioservice           0x7200000000000001:0x39         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@21001
    [started]  ioservice           0x7200000000000001:0x3a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@21002
    [started]  confd               0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x3c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0x3d         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0x3e         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0x3f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22002
Devices:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1583
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1591
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1583
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1590
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1592
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1590
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1592
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1591
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf

Happy path IOs:

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 mb s3://test --endpoint-url http://$IP:$Port
make_bucket: test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-18 03:24:46 test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-18 03:25:06 1048576000 file_1gb

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 0,
    "healthy": 1092266662
  }
}

Degraded IOs: Node failure and degraded write test

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get deployment cortx-data-ssc-vm-g2-rhev4-1632 -o yaml > cortx-data-ssc-vm-g2-rhev4-1632.yaml
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl delete deployment cortx-data-ssc-vm-g2-rhev4-1632
deployment.apps "cortx-data-ssc-vm-g2-rhev4-1632" deleted

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d | grep offline
    [offline]  hax                 0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [offline]  ioservice           0x7200000000000001:0x49         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [offline]  ioservice           0x7200000000000001:0x4a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [offline]  confd               0x7200000000000001:0x4b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22002
    [offline]  /dev/sdd
    [offline]  /dev/sde
    [offline]  /dev/sdc
    [offline]  /dev/sdg
    [offline]  /dev/sdh
    [offline]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 1092266662,
    "healthy": 0
  }
}

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb1 --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb1
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-18 03:25:06 1048576000 file_1gb
2022-07-18 03:43:46 1048576000 file_1gb1

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 2139095034,
    "healthy": 0
  }
}

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl apply -f cortx-data-ssc-vm-g2-rhev4-1632.yaml
deployment.apps/cortx-data-ssc-vm-g2-rhev4-1632 created

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 2139095034
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3170@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x40         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x41         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x42         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x43         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x44         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  ioservice           0x7200000000000001:0x45         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21001
    [started]  ioservice           0x7200000000000001:0x46         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@21002
    [started]  confd               0x7200000000000001:0x47         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3156@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x49         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x4b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x4c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x4d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0x4e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x50         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x51         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0x53         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1583
    [started]  hax                 0x7200000000000001:0x54         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@22001
    [started]  ioservice           0x7200000000000001:0x55         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@21001
    [started]  ioservice           0x7200000000000001:0x56         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@21002
    [started]  confd               0x7200000000000001:0x57         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1583@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x58         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0x59         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x5a         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x5b         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x5c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x5d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x5e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22002
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x60         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw_s3              0x7200000000000001:0x61         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1591
    [started]  hax                 0x7200000000000001:0x62         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1591@22001
    [started]  rgw_s3              0x7200000000000001:0x63         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1591@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1583
    [started]  hax                 0x7200000000000001:0x64         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1583@22001
    [started]  rgw_s3              0x7200000000000001:0x65         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1583@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x66         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw_s3              0x7200000000000001:0x67         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x68         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw_s3              0x7200000000000001:0x69         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    [started]  hax                 0x7200000000000001:0x6a         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  rgw_s3              0x7200000000000001:0x6b         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3031@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x6c         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw_s3              0x7200000000000001:0x6d         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1590
    [started]  hax                 0x7200000000000001:0x6e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1590@22001
    [started]  rgw_s3              0x7200000000000001:0x6f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1590@21001
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3031@22002
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x70         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw_s3              0x7200000000000001:0x71         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    [started]  hax                 0x7200000000000001:0x72         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@22001
    [started]  rgw_s3              0x7200000000000001:0x73         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3156@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x74         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw_s3              0x7200000000000001:0x75         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x76         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw_s3              0x7200000000000001:0x77         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0x78         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  rgw_s3              0x7200000000000001:0x79         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3158@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x7a         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw_s3              0x7200000000000001:0x7b         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    [started]  hax                 0x7200000000000001:0x7c         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@22001
    [started]  rgw_s3              0x7200000000000001:0x7d         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3170@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1592
    [started]  hax                 0x7200000000000001:0x7e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1592@22001
    [started]  rgw_s3              0x7200000000000001:0x7f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1592@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x80         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw_s3              0x7200000000000001:0x81         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x82         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  rgw_s3              0x7200000000000001:0x83         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3160@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x84         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  rgw_s3              0x7200000000000001:0x85         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-3157@21001
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x86         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw_s3              0x7200000000000001:0x87         inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x88         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  rgw_s3              0x7200000000000001:0x89         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x8a         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw_s3              0x7200000000000001:0x8b         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x8c         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw_s3              0x7200000000000001:0x8d         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21001
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x8e         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw_s3              0x7200000000000001:0x8f         inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21001
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3158@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283  (RC)
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3160@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1590
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1590@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22001
    [started]  ioservice           0x7200000000000001:0x29         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@21002
    [started]  confd               0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-3157@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x2d         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x2f         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1592
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@22001
    [started]  ioservice           0x7200000000000001:0x31         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@21001
    [started]  ioservice           0x7200000000000001:0x32         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@21002
    [started]  confd               0x7200000000000001:0x33         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1592@22002
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0x35         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0x36         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0x37         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1591
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@22001
    [started]  ioservice           0x7200000000000001:0x39         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@21001
    [started]  ioservice           0x7200000000000001:0x3a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@21002
    [started]  confd               0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1591@22002
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x3c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0x3d         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0x3e         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0x3f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22002
Devices:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3170
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3156
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1583
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1591
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1583
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3031
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1590
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3031
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3156
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3158
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3170
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1592
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3160
    cortx-server-headless-svc-ssc-vm-g2-rhev4-3157
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3158
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3160
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1590
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-3157
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1592
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1591
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb2 --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb2
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-18 03:25:06 1048576000 file_1gb
2022-07-18 03:43:46 1048576000 file_1gb1
2022-07-18 04:05:02 1048576000 file_1gb2

[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 0,
    "healthy": 3231361705
  }
}

Node restarts tests:

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl delete pod cortx-data-ssc-vm-g2-rhev4-1635-6dd69f4fdc-z94lz
pod "cortx-data-ssc-vm-g2-rhev4-1635-6dd69f4fdc-z94lz" deleted
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-18 03:24:46 test
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-18 03:25:06 1048576000 file_1gb
2022-07-18 03:43:46 1048576000 file_1gb1
2022-07-18 04:05:02 1048576000 file_1gb2
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Write file..."
Write file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb3 --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb3
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-18 03:25:06 1048576000 file_1gb
2022-07-18 03:43:46 1048576000 file_1gb1
2022-07-18 04:05:02 1048576000 file_1gb2
2022-07-18 04:28:17 1048576000 file_1gb3
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Read file..."
Read file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp s3://test/file_1gb file_1gbDn --endpoint-url http://$IP:$Port
download: s3://test/file_1gb to ./file_1gbDn
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp s3://test/file_1gb1 file_1gbDn1 --endpoint-url http://$IP:$Port
download: s3://test/file_1gb1 to ./file_1gbDn1
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp s3://test/file_1gb2 file_1gbDn2 --endpoint-url http://$IP:$Port
download: s3://test/file_1gb2 to ./file_1gbDn2

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1631-648c5684c5-nmwzw -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1631 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 0,
    "healthy": 4279937698
  }
}

Also ran hare-sanity job on 25N deployed setup at https://eos-jenkins.colo.seagate.com/job/Hare/job/Hare_k8s_Sanity_PR/306/ but it was failed at setup stage for test_degraded_iteration_write_read_partial_delete

14:01:28  ==================================== ERRORS ====================================
14:01:28  _ ERROR at setup of TestIOWorkloadDegradedPath.test_degraded_iteration_write_read_partial_delete _
14:01:28  
14:01:28  self = <iostability.test_iostability_degraded_path.TestIOWorkloadDegradedPath object at 0x7ff03fce1650>
14:01:28  
14:01:28      def setup_method(self):
14:01:28          """Setup Method"""
14:01:28          self.log.info("Setup Method Started")
14:01:28          self.log.info("Start Procpath collection")
14:01:28          self.proc_path = EnableProcPathStatsCollection(CMN_CFG)
14:01:28          self.log_collect = ServerOSLogsCollectLib(CMN_CFG)
14:01:28          resp = self.proc_path.setup_requirement()
14:01:28          assert_utils.assert_true(resp[0], resp[1])
14:01:28          self.proc_path.start_collection()
14:01:28          time.sleep(30)
14:01:28          resp = self.proc_path.validate_collection()
14:01:28  >       assert_utils.assert_true(resp[0], resp[1])
14:01:28  
14:01:28  tests/iostability/test_iostability_degraded_path.py:99: 
14:01:28  _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
14:01:28  
14:01:28  matcher = False, reason = 'process is not alive'
14:01:28  
14:01:28      def assert_true(matcher, reason=""):
14:01:28          """AssertTrue Implementation."""
14:01:28  >       assert matcher, reason if reason else matcher
14:01:28  E       AssertionError: process is not alive
14:01:28  
14:01:28  commons/utils/assert_utils.py:209: AssertionError
14:01:28  ---------------------------- Captured stderr setup -----------------------------
14:01:28  Process Process-2:
14:01:28  Traceback (most recent call last):
14:01:28    File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
14:01:28      self.run()
14:01:28    File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
14:01:28      self._target(*self._args, **self._kwargs)
14:01:28    File "/root/workspace/Hare_k8s_Sanity_PR/libs/dtm/ProcPathStasCollection.py", line 66, in exe_cmd
14:01:28      worker.execute_cmd(cmd=cmd)
14:01:28    File "/root/workspace/Hare_k8s_Sanity_PR/commons/helpers/host.py", line 220, in execute_cmd
14:01:28      raise IOError(err)
14:01:28  OSError: ['___   ___   ___   __    ___    __   _____  _', '| |_) | |_) / / \\ / /`  | |_)  / /\\   | |  | |_|', '|_|   |_| \\ \\_\\_/ \\_\\_, |_|   /_/--\\  |_|  |_| |', '', 'a process tree analysis workbench', '', 'usage: procpath record [-h] [-f PROCFILE-LIST] [-e ENVIRONMENT] -d DATABASE-FILE', '[-p PID-LIST] [-i INTERVAL] [-r RECNUM] [-v REEVALNUM]', '[--stop-without-result]', '[query]', 'procpath record: error: argument -p/--pid-list: expected one argument']
14:01:28  Process Process-5:
14:01:28  Traceback (most recent call last):
14:01:28    File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
14:01:28      self.run()
14:01:28    File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
14:01:28      self._target(*self._args, **self._kwargs)
14:01:28    File "/root/workspace/Hare_k8s_Sanity_PR/libs/dtm/ProcPathStasCollection.py", line 66, in exe_cmd
14:01:28      worker.execute_cmd(cmd=cmd)
14:01:28    File "/root/workspace/Hare_k8s_Sanity_PR/commons/helpers/host.py", line 220, in execute_cmd
14:01:28      raise IOError(err)
14:01:28  OSError: ['___   ___   ___   __    ___    __   _____  _', '| |_) | |_) / / \\ / /`  | |_)  / /\\   | |  | |_|', '|_|   |_| \\ \\_\\_/ \\_\\_, |_|   /_/--\\  |_|  |_| |', '', 'a process tree analysis workbench', '', 'usage: procpath record [-h] [-f PROCFILE-LIST] [-e ENVIRONMENT] -d DATABASE-FILE', '[-p PID-LIST] [-i INTERVAL] [-r RECNUM] [-v REEVALNUM]', '[--stop-without-result]', '[query]', 'procpath record: error: argument -p/--pid-list: expected one argument']
14:01:28  Process Process-3:
14:01:28  Traceback (most recent call last):
14:01:28    File "/usr/local/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
14:01:28      self.run()
14:01:28    File "/usr/local/lib/python3.7/multiprocessing/process.py", line 99, in run
14:01:28      self._target(*self._args, **self._kwargs)
14:01:28    File "/root/workspace/Hare_k8s_Sanity_PR/libs/dtm/ProcPathStasCollection.py", line 66, in exe_cmd
14:01:28      worker.execute_cmd(cmd=cmd)
14:01:28    File "/root/workspace/Hare_k8s_Sanity_PR/commons/helpers/host.py", line 220, in execute_cmd
14:01:28      raise IOError(err)

cc. @vaibhavparatwar, @mssawant

mssawant commented 2 years ago

retest this please.

mssawant commented 2 years ago

https://jts.seagate.com/browse/CORTX-33634 from Motr side is required for IOs to succeed post node restart. Above build with which the testing was done includes a patch from @yeshpal-jain-seagate that enables successful IO post node restart. cc @madhavemuri

supriyachavan4398 commented 2 years ago

HappyPath-Type1 Total Client:-2 Build:- :2.0.0-7261 Test run hour: ~18 Hour Ref. https://eos-jenkins.colo.seagate.com/job/QA/job/IOStabilityTestRuns/301 It's successful without any error. Console output is attached here CORTX_33034consoleText.txt cc. @vaibhavparatwar, @mssawant

yeshpal-jain-seagate commented 2 years ago

https://jts.seagate.com/browse/CORTX-33634 from Motr side is required for IOs to succeed post node restart. Above build with which the testing was done includes a patch from @yeshpal-jain-seagate that enables successful IO post node restart. cc @madhavemuri

@mssawant CORTX-33634 is under debug which is a regression after 842 build, this patch is helping to fix container restart and making container failures more stable which will help us to run happy path IO for longer duration(which would fix CORTX-33418 and CORTX-33034), this patch can be tested only with the container restart. With test_hare_fix branch which has reverted 68efc9a5(PR#1930) and this PR from hare is been tested with pod restart for multiple iteration and that works fine, so once we have a fix for CORTX-33634, without any in hax, pod restart should work.

vaibhavparatwar commented 2 years ago

retest this please

mssawant commented 2 years ago

retest this please

mssawant commented 2 years ago

retest this please.

mssawant commented 2 years ago

re-tested, 6N deployment, regular and degraded IO,

REPOSITORY                                            TAG                    IMAGE ID       CREATED             SIZE
cortx-docker.colo.seagate.com/seagate/cortx-rgw       2.0.0-7283-custom-ci   9eca43336894   About an hour ago   1.43GB
cortx-docker.colo.seagate.com/seagate/cortx-data      2.0.0-7283-custom-ci   ef24bd0dd685   About an hour ago   1.05GB
cortx-docker.colo.seagate.com/seagate/cortx-control   2.0.0-7283-custom-ci   751309f678c4   About an hour ago   830MB
k8s.gcr.io/kube-apiserver                             v1.22.12               d922296a8b85   6 days ago          128MB
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712  (RC)
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  rgw_s3              0x7200000000000001:0x19         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x1a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  rgw_s3              0x7200000000000001:0x1b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  rgw_s3              0x7200000000000001:0x1d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x1e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  rgw_s3              0x7200000000000001:0x1f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  rgw_s3              0x7200000000000001:0x21         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x22         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  rgw_s3              0x7200000000000001:0x23         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@21001
Devices:
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]#
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 mb s3://test
make_bucket: test
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gb s3://test
upload: ./file_1gb to s3://test/file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-19 11:12:35 1048576000 file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# kubectl exec -it cortx-data-ssc-vm-g4-rhev4-0710-5cc7857dbb-lbp98 -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 1048576000
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
[root@ssc-vm-g4-rhev4-0710 ~]# kubectl exec -it cortx-data-ssc-vm-g4-rhev4-0710-5cc7857dbb-lbp98 -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# consul kv get -recurse bq
bq-delivered/cortx-data-headless-svc-ssc-vm-g4-rhev4-0710:112
bq-delivered/cortx-data-headless-svc-ssc-vm-g4-rhev4-0711:112
bq-delivered/cortx-data-headless-svc-ssc-vm-g4-rhev4-0712:112
bq-delivered/cortx-data-headless-svc-ssc-vm-g4-rhev4-1765:111
bq-delivered/cortx-data-headless-svc-ssc-vm-g4-rhev4-1766:112
bq-delivered/cortx-data-headless-svc-ssc-vm-g4-rhev4-1767:112
bq-delivered/cortx-server-headless-svc-ssc-vm-g4-rhev4-0710:112
bq-delivered/cortx-server-headless-svc-ssc-vm-g4-rhev4-0711:112
bq-delivered/cortx-server-headless-svc-ssc-vm-g4-rhev4-0712:112
bq-delivered/cortx-server-headless-svc-ssc-vm-g4-rhev4-1765:112
bq-delivered/cortx-server-headless-svc-ssc-vm-g4-rhev4-1766:112
bq-delivered/cortx-server-headless-svc-ssc-vm-g4-rhev4-1767:112
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-19 11:12:35 1048576000 file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 mb s3://test1
make_bucket: test1
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 mb s3://test2
make_bucket: test2
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 mb s3://test3
make_bucket: test3
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1
file_10mb    file_1gb     file_1gbDn   file_1gbDn1
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gbDn s3://test1
upload: ./file_1gbDn to s3://test1/file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gbDn s3://test2
upload: ./file_1gbDn to s3://test2/file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gbDn s3://test3
upload: ./file_1gbDn to s3://test3/file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test
2022-07-19 11:12:35 1048576000 file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test1
2022-07-19 11:18:27 1048576000 file_1gbDn
^[[A[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test2
^[[A2022-07-19 11:18:45 1048576000 file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test3
2022-07-19 11:19:00 1048576000 file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# kubectl exec -it cortx-data-ssc-vm-g4-rhev4-0710-5cc7857dbb-lbp98 -c cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# hctl status
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 1048576000
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [offline]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [offline]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21001
    [offline]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21002
    [offline]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712  (RC)
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  rgw_s3              0x7200000000000001:0x19         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x1a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  rgw_s3              0x7200000000000001:0x1b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  rgw_s3              0x7200000000000001:0x1d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x1e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  rgw_s3              0x7200000000000001:0x1f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  rgw_s3              0x7200000000000001:0x21         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x22         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  rgw_s3              0x7200000000000001:0x23         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@21001
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]#

Sequence of events for failed node's processes,

[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# consul kv get -recurse bq | egrep '0x7200000000000001:0x0|0x7200000000000001:0x1|0x7200000000000001:0x2|0x7200000000000001:0x3'
bq/100:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x23", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/101:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x19", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/102:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x19", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/103:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1b", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/104:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1b", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/105:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1f", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/106:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1f", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/107:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/108:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/109:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/11:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/110:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/111:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/112:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/113:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/114:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/115:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/116:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/117:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/118:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/119:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/120:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/121:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/122:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/123:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/124:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/125:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/126:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/127:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/128:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/129:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/130:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/131:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/132:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/133:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/134:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/135:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/136:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/137:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/138:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/139:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/14:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/140:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/141:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/16:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/19:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/2:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/20:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/22:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/23:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/26:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/27:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/28:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/3:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x0", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/30:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/39:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x14", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/4:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x14", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/40:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x17", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/41:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x16", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/42:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x15", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/43:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x14", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/44:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x3", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/46:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/49:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/5:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x14", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/50:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x17", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/51:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/55:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/56:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/59:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/60:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/62:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/65:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x2", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/69:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x16", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/7:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/71:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x16", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/72:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x15", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/73:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x15", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/74:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/75:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/76:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/77:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_STOPPING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/78:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/79:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x22", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/80:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1c", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/81:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x22", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/82:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x23", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/83:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1d", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/84:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x18", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/85:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1c", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/86:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x19", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/87:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x18", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/88:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1a", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/89:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1e", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/90:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1b", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/91:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1a", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/92:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1e", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/93:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1f", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/94:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x20", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/95:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x20", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/96:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x21", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/97:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1d", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/98:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x23", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
bq/99:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x1d", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_OTHER"}}
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]#
mssawant commented 2 years ago

Node restart,

[root@ssc-vm-g4-rhev4-0710 ~]# kubectl delete pod cortx-data-ssc-vm-g4-rhev4-1766-dd4c8f9f4-xwgnz
pod "cortx-data-ssc-vm-g4-rhev4-1766-dd4c8f9f4-xwgnz" deleted
[root@ssc-vm-g4-rhev4-0710 ~]#

Sequence of events for restarted events,

[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# consul kv get -recurse bq | egrep '0x7200000000000001:0x10|0x7200000000000001:0x11|0x7200000000000001:0x12|0x7200000000000001:0x13'
bq/11:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/126:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/127:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/128:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/129:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/130:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/131:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/132:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STOPPED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/133:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/134:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/135:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/136:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/137:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/138:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/139:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/14:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/140:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/141:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/19:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_HA"}}
bq/20:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/22:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/23:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/46:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x13", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/49:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/55:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_STARTED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/56:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x11", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/62:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x12", "state": "M0_CONF_HA_PROCESS_DTM_RECOVERED", "type": "M0_CONF_HA_PROCESS_M0D"}}
bq/7:{"message_type": "PROCESS-STATE-UPDATE", "payload": {"fid": "0x7200000000000001:0x10", "state": "M0_CONF_HA_PROCESS_STARTING", "type": "M0_CONF_HA_PROCESS_HA"}}
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]#
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls
2022-07-19 11:12:11 test
2022-07-19 11:17:52 test1
2022-07-19 11:17:54 test2
2022-07-19 11:17:57 test3
2022-07-19 11:22:55 test4
2022-07-19 11:37:55 test5
[root@ssc-vm-g4-rhev4-0710 ~]#
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# hctl status
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 5730118314
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
mssawant commented 2 years ago

Degraded byte count on node failure,

[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]# hctl status
Bytecount:
    critical : 0
    damaged : 0
    degraded : 5730118314
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712  (RC)
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0710@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [offline]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [offline]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21001
    [offline]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21002
    [offline]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  rgw_s3              0x7200000000000001:0x19         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0710
    [started]  hax                 0x7200000000000001:0x1a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@22001
    [started]  rgw_s3              0x7200000000000001:0x1b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0710@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  rgw_s3              0x7200000000000001:0x1d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x1e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  rgw_s3              0x7200000000000001:0x1f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  rgw_s3              0x7200000000000001:0x21         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x22         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  rgw_s3              0x7200000000000001:0x23         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@21001
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0710 /]#
mssawant commented 2 years ago

Above testing is done using Motr branch https://github.com/yeshpal-jain-seagate/cortx-motr/tree/test_hare_fix. Hare Sanity is failing in degraded IO test for some reason, although manual testing of degraded IO succeeded. This can be debugged separately. Going ahead with merging the PR.

mssawant commented 2 years ago

Fixed Hare sanity.