search

Seagate / cortx-hare

CORTX Hare configures Motr object store, starts/stops Motr services, and notifies Motr of service and device faults.
https://github.com/Seagate/cortx
Apache License 2.0
13 stars 80 forks source link

ha: process state update is not a broadcast #2032

Closed mssawant closed 2 years ago

mssawant commented 2 years ago

It is possible that Consul may not send service state notifications periodically as configured, see https://github.com/hashicorp/consul/issues/7069. Sometimes the notification is sent only on Consul service state change. During bootstrap if a peer process has not started, Hare reports it as offline in nvec reply to another process. Relying on Consul notification, Hare expects that the offline state will be updated to online on receiving a corresponding periodical Consul notification. But as the notification might get delayed or not arrive at all, the offline state of the process may not be updated to online, which may further lead to inconsistencies.

Solution: Watch processes/ key value for the updates reported by the individual process and accordingly broadcast the process state change to peer motr processes.

Signed-off-by: Mandar Sawant mandar.sawant@seagate.com

mssawant commented 2 years ago

sns repair completed successfully in non-container environment,

root@ssc-vm-g4-rhev4-0786 ~ > hctl bootstrap --mkfs 1ios3node.yaml
2022-03-18 17:18:25: Generating cluster configuration... OK
2022-03-18 17:18:29: Starting Consul server on this node......... OK
2022-03-18 17:18:35: Importing configuration into the KV store... OK
2022-03-18 17:18:36: Starting Consul on other nodes...Consul ready on all nodes
2022-03-18 17:18:38: Updating Consul configuraton from the KV store... OK
2022-03-18 17:18:42: Waiting for the RC Leader to get elected... OK
2022-03-18 17:18:42: Starting Motr (phase1, mkfs)... OK
2022-03-18 17:18:50: Starting Motr (phase1, m0d)... OK
2022-03-18 17:18:54: Starting Motr (phase2, mkfs)... OK
2022-03-18 17:19:05: Starting Motr (phase2, m0d)... OK
2022-03-18 17:19:10: Checking health of services... OK
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x63 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x92 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x27         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:10.230.247.111@21002
    [unknown]  m0_client_other     0x7200000000000001:0x39         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x3c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x6          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0x9          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xc          inet:tcp:10.230.250.52@21002
    [unknown]  m0_client_other     0x7200000000000001:0x1b         inet:tcp:10.230.250.52@21501
    [unknown]  m0_client_other     0x7200000000000001:0x1e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x42         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x45         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x48         inet:tcp:10.230.240.142@21002
    [unknown]  m0_client_other     0x7200000000000001:0x57         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdb
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
root@ssc-vm-g4-rhev4-0786 ~ > hctl drive-state --json $(jq --null-input --compact-output '{ node: "ssc-vm-g4-rhev4-0786.colo.seagate.com", source_type: "drive", device: "/dev/sdd", state: "failed" }')
0
2022-03-18 17:19:24,392 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7feda2620080>
2022-03-18 17:19:24,392 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7feda5b64510>
2022-03-18 17:19:24,396 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-18 17:19:24,398 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 91
2022-03-18 17:19:24,408 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-18 17:19:24,409 [INFO] Written to epoch: 2
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x63 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x92 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x27         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:10.230.247.111@21002
    [unknown]  m0_client_other     0x7200000000000001:0x39         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x3c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x6          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0x9          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xc          inet:tcp:10.230.250.52@21002
    [unknown]  m0_client_other     0x7200000000000001:0x1b         inet:tcp:10.230.250.52@21501
    [unknown]  m0_client_other     0x7200000000000001:0x1e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x42         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x45         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x48         inet:tcp:10.230.240.142@21002
    [unknown]  m0_client_other     0x7200000000000001:0x57         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdb
    [failed]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
root@ssc-vm-g4-rhev4-0786 ~ > hctl drive-state --json $(jq --null-input --compact-output '{ node: "ssc-vm-g4-rhev4-0786.colo.seagate.com", source_type: "drive", device: "/dev/sdd", state: "repair" }')
0
2022-03-18 17:19:34,788 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7f30e8f890f0>
2022-03-18 17:19:34,788 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7f30ec4cd510>
2022-03-18 17:19:34,792 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-18 17:19:34,794 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 93
2022-03-18 17:19:34,804 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-18 17:19:34,804 [INFO] Written to epoch: 3
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x63 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x92 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x27         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:10.230.247.111@21002
    [unknown]  m0_client_other     0x7200000000000001:0x39         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x3c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x6          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0x9          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xc          inet:tcp:10.230.250.52@21002
    [unknown]  m0_client_other     0x7200000000000001:0x1b         inet:tcp:10.230.250.52@21501
    [unknown]  m0_client_other     0x7200000000000001:0x1e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x42         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x45         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x48         inet:tcp:10.230.240.142@21002
    [unknown]  m0_client_other     0x7200000000000001:0x57         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdb
    [repairing]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
root@ssc-vm-g4-rhev4-0786 ~ > hctl repair start
2022-03-18 17:19:41,655 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7fb767ca80f0>
2022-03-18 17:19:41,655 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7fb76b1ec510>
2022-03-18 17:19:41,659 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-18 17:19:41,660 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 93
2022-03-18 17:19:41,667 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-18 17:19:41,668 [INFO] Written to epoch: 4
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x63 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x92 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x27         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x2a         inet:tcp:10.230.247.111@21002
    [unknown]  m0_client_other     0x7200000000000001:0x39         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x3c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x6          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0x9          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xc          inet:tcp:10.230.250.52@21002
    [unknown]  m0_client_other     0x7200000000000001:0x1b         inet:tcp:10.230.250.52@21501
    [unknown]  m0_client_other     0x7200000000000001:0x1e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x42         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x45         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x48         inet:tcp:10.230.240.142@21002
    [unknown]  m0_client_other     0x7200000000000001:0x57         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdb
    [repaired]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
root@ssc-vm-g4-rhev4-0786 ~ >
nkommuri commented 2 years ago

Tested on my 3 node non-k8s cluster. its working.

mssawant commented 2 years ago

Degraded IO post failure, repair, repaired and regular IO post rebalancing.

root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x99 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0xe0 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0xa          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:10.230.250.52@21002
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:10.230.250.52@21003
    [unknown]  m0_client_other     0x7200000000000001:0x2b         inet:tcp:10.230.250.52@21501
    [unknown]  m0_client_other     0x7200000000000001:0x2e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x35         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x38         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:10.230.247.111@21002
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:10.230.247.111@21003
    [unknown]  m0_client_other     0x7200000000000001:0x59         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x63         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x66         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:10.230.240.142@21002
    [started]  ioservice           0x7200000000000001:0x78         inet:tcp:10.230.240.142@21003
    [unknown]  m0_client_other     0x7200000000000001:0x87         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x8a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
root@ssc-vm-g4-rhev4-0786 ~ > m0client -l 'inet:tcp:10.230.250.52@21502' -H 'inet:tcp:10.230.250.52@22001' -p '<0x7000000000000001:0xe0>' -P '<0x7200000000000001:0x2e>'
m0client >>write 1048680 /tmp/128M 4096 200 50
m0client >>read 1048680 /tmp/read_1048680_regular 4096 200 50
m0client >>quit
Help:
touch   OBJ_ID
write   OBJ_ID SRC_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO UPDATE_FLAG OFFSET
read    OBJ_ID DEST_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO
delete  OBJ_ID
help
quit
root@ssc-vm-g4-rhev4-0786 ~ > ls -l /tmp/read_1048680_regular
-rw-r--r-- 1 root root 819200 Mar 21 14:27 /tmp/read_1048680_regular
root@ssc-vm-g4-rhev4-0786 ~ >

root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x99 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0xe0 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0xa          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:10.230.250.52@21002
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:10.230.250.52@21003
    [unknown]  m0_client_other     0x7200000000000001:0x2b         inet:tcp:10.230.250.52@21501
    [offline]  m0_client_other     0x7200000000000001:0x2e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x35         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x38         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:10.230.247.111@21002
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:10.230.247.111@21003
    [unknown]  m0_client_other     0x7200000000000001:0x59         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x63         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x66         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:10.230.240.142@21002
    [started]  ioservice           0x7200000000000001:0x78         inet:tcp:10.230.240.142@21003
    [unknown]  m0_client_other     0x7200000000000001:0x87         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x8a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [failed]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
root@ssc-vm-g4-rhev4-0786 ~ > m0client -l 'inet:tcp:10.230.250.52@21502' -H 'inet:tcp:10.230.250.52@22001' -p '<0x7000000000000001:0xe0>' -P '<0x7200000000000001:0x2e>'
m0client >>read 1048680 /tmp/read_1048680_degraded_after_failure 4096 200 50
motr[443818]:  b9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x55e8578c2ea0 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[443818]:  ba70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x55e8578c2ea0 dgmap_nr=3
motr[443818]:  d9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x55e8578df010 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[443818]:  da70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x55e8578df010 dgmap_nr=3
motr[443818]:  b9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x55e8578df010 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[443818]:  ba70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x55e8578df010 dgmap_nr=2
motr[443818]:  d9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x55e8578df010 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[443818]:  da70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x55e8578df010 dgmap_nr=3
m0client >>quit
Help:
touch   OBJ_ID
write   OBJ_ID SRC_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO UPDATE_FLAG OFFSET
read    OBJ_ID DEST_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO
delete  OBJ_ID
help
quit
root@ssc-vm-g4-rhev4-0786 ~ > ls -l /tmp/read_1048680_degra
read_1048680_degraded_after_failure          read_1048680_degraded_disk_fail              read_1048680_degraded_disk_rebalance_online  read_1048680_degraded_disk_repair
root@ssc-vm-g4-rhev4-0786 ~ > ls -l /tmp/read_1048680_degraded_after_failure
-rw-r--r-- 1 root root 819200 Mar 21 14:30 /tmp/read_1048680_degraded_after_failure
root@ssc-vm-g4-rhev4-0786 ~ >

root@ssc-vm-g4-rhev4-0786 ~ > hctl drive-state --json $(jq --null-input --compact-output '{ node: "ssc-vm-g4-rhev4-0787.colo.seagate.com", source_type: "drive", device: "/dev/sdd", state: "repair" }')
0
2022-03-21 14:31:31,769 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7f13e52d2128>
2022-03-21 14:31:31,769 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7f13e8817510>
2022-03-21 14:31:31,773 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-21 14:31:31,774 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 93
2022-03-21 14:31:31,786 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-21 14:31:31,786 [INFO] Written to epoch: 3
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x99 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0xe0 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0xa          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:10.230.250.52@21002
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:10.230.250.52@21003
    [unknown]  m0_client_other     0x7200000000000001:0x2b         inet:tcp:10.230.250.52@21501
    [offline]  m0_client_other     0x7200000000000001:0x2e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x35         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x38         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:10.230.247.111@21002
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:10.230.247.111@21003
    [unknown]  m0_client_other     0x7200000000000001:0x59         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x63         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x66         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:10.230.240.142@21002
    [started]  ioservice           0x7200000000000001:0x78         inet:tcp:10.230.240.142@21003
    [unknown]  m0_client_other     0x7200000000000001:0x87         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x8a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [repairing]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
root@ssc-vm-g4-rhev4-0786 ~ > m0client -l 'inet:tcp:10.230.250.52@21502' -H 'inet:tcp:10.230.250.52@22001' -p '<0x7000000000000001:0xe0>' -P '<0x7200000000000001:0x2e>'
motr[447231]:  75a0  ERROR  [pool/pool_machine.c:738:m0_poolmach_state_transit]  <7600000000000001:9a>: This pool is in DUD state; event_index=7 event_state=4
m0client >>read 1048680 /tmp/read_1048680_degraded_after_repairing 4096 200 50
motr[447231]:  c9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x561f141093d0 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[447231]:  ca70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x561f141093d0 dgmap_nr=3
motr[447231]:  a9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x561f1412d010 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[447231]:  aa70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x561f1412d010 dgmap_nr=3
motr[447231]:  d9f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x561f14129010 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[447231]:  da70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x561f14129010 dgmap_nr=2
motr[447231]:  79f0  ERROR  [io_req_fop.c:363:io_bottom_half]  ioo=0x561f1412d010 from=inet:tcp:10.230.247.111@21002 rc=-5 ti_rc=-5 @<4300000700000000:100068>
motr[447231]:  7a70   WARN  [io_req.c:1665:ioreq_dgmode_read]  Process failed parity groups in dgmode/read ioo=0x561f1412d010 dgmap_nr=3
m0client >>quit
Help:
touch   OBJ_ID
write   OBJ_ID SRC_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO UPDATE_FLAG OFFSET
read    OBJ_ID DEST_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO
delete  OBJ_ID
help
quit
root@ssc-vm-g4-rhev4-0786 ~ > ls -l /tmp/read_1048680_degraded_after_repairing
-rw-r--r-- 1 root root 819200 Mar 21 14:32 /tmp/read_1048680_degraded_after_repairing
root@ssc-vm-g4-rhev4-0786 ~ >
root@ssc-vm-g4-rhev4-0786 ~ > hctl drive-state --json $(jq --null-input --compact-output '{ node: "ssc-vm-g4-rhev4-0787.colo.seagate.com", source_type: "drive", device: "/dev/sdd", state: "repair" }')
0
2022-03-21 15:38:32,680 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7f54f574f0f0>
2022-03-21 15:38:32,681 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7f54f8c94510>
2022-03-21 15:38:32,684 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-21 15:38:32,686 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 93
2022-03-21 15:38:32,695 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-21 15:38:32,696 [INFO] Written to epoch: 3
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x99 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0xe0 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0xa          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:10.230.250.52@21002
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:10.230.250.52@21003
    [unknown]  m0_client_other     0x7200000000000001:0x2b         inet:tcp:10.230.250.52@21501
    [started]  m0_client_other     0x7200000000000001:0x2e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x35         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x38         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:10.230.247.111@21002
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:10.230.247.111@21003
    [unknown]  m0_client_other     0x7200000000000001:0x59         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x63         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x66         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:10.230.240.142@21002
    [started]  ioservice           0x7200000000000001:0x78         inet:tcp:10.230.240.142@21003
    [unknown]  m0_client_other     0x7200000000000001:0x87         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x8a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [repairing]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
root@ssc-vm-g4-rhev4-0786 ~ >
root@ssc-vm-g4-rhev4-0786 ~ > hctl repair start
2022-03-21 15:40:32,707 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7f0bc2fc10f0>
2022-03-21 15:40:32,707 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7f0bc6506510>
2022-03-21 15:40:32,711 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-21 15:40:32,712 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 93
2022-03-21 15:40:32,722 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-21 15:40:32,722 [INFO] Written to epoch: 4
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x99 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0xe0 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0xa          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:10.230.250.52@21002
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:10.230.250.52@21003
    [unknown]  m0_client_other     0x7200000000000001:0x2b         inet:tcp:10.230.250.52@21501
    [started]  m0_client_other     0x7200000000000001:0x2e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x35         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x38         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:10.230.247.111@21002
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:10.230.247.111@21003
    [unknown]  m0_client_other     0x7200000000000001:0x59         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x63         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x66         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:10.230.240.142@21002
    [started]  ioservice           0x7200000000000001:0x78         inet:tcp:10.230.240.142@21003
    [unknown]  m0_client_other     0x7200000000000001:0x87         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x8a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [repaired]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
(failed reverse-i-search)`rebalance ': hctl ^Cbalance start
root@ssc-vm-g4-rhev4-0786 ~ >
...
Mar 21 15:31:16 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  8dd0   WARN  [setup.c:1140:cs_service_init]  Service M0_CST_FDMI (<7300000000000001:1b>) is already >Mar 21 15:31:17 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: Started
Mar 21 15:31:17 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: lt-m0d: systemd READY notification successful
Mar 21 15:31:17 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: Press CTRL+C to quit.
Mar 21 15:31:17 ssc-vm-g4-rhev4-0786.colo.seagate.com systemd[1]: Started m0d helper for 0x7200000000000001:0xd service-FID.
Mar 21 15:40:33 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  ca70  ERROR  [cm/sw_update_fom.c:122:cm_swu_fom_tick]  SWU phase=0 rc=1.
Mar 21 15:40:33 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  ca70  ERROR  [cm/sw_update_fom.c:122:cm_swu_fom_tick]  SWU phase=0 rc=-61.
Mar 21 15:40:33 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  d840   WARN  [sns/cm/cm.c:730:m0_sns_cm_stop]  Time: 356215849 Read Size: 57344 Write size: 4096
Mar 21 15:49:03 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  ea70  ERROR  [cm/sw_update_fom.c:122:cm_swu_fom_tick]  SWU phase=0 rc=1.
Mar 21 15:49:03 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  ea70  ERROR  [cm/sw_update_fom.c:122:cm_swu_fom_tick]  SWU phase=0 rc=-61.
Mar 21 15:49:03 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  9850   WARN  [sns/cm/cm.c:730:m0_sns_cm_stop]  Time: 235480508 Read Size: 4096 Write size: 0
Mar 21 16:05:51 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  ca70  ERROR  [cm/sw_update_fom.c:122:cm_swu_fom_tick]  SWU phase=0 rc=1.
Mar 21 16:05:51 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  ca70  ERROR  [cm/sw_update_fom.c:122:cm_swu_fom_tick]  SWU phase=0 rc=-61.
Mar 21 16:05:51 ssc-vm-g4-rhev4-0786.colo.seagate.com motr-server[531411]: motr[531411]:  d840   WARN  [sns/cm/cm.c:730:m0_sns_cm_stop]  Time: 268058533 Read Size: 57344 Write size: 4096
root@ssc-vm-g4-rhev4-0786 ~ > hctl rebalance start
2022-03-21 15:49:02,763 [DEBUG] Bound <class 'hax.common.HaxGlobalState'> to an instance <hax.common.HaxGlobalState object at 0x7f27312ee0f0>
2022-03-21 15:49:02,763 [DEBUG] Created and configured an injector, config=<function di_configuration at 0x7f2734832510>
2022-03-21 15:49:02,771 [DEBUG] Starting new HTTP connection (1): 127.0.0.1:8500
2022-03-21 15:49:02,773 [DEBUG] http://127.0.0.1:8500 "GET /v1/kv/eq-epoch HTTP/1.1" 200 93
2022-03-21 15:49:02,784 [DEBUG] http://127.0.0.1:8500 "PUT /v1/txn HTTP/1.1" 200 219
2022-03-21 15:49:02,784 [INFO] Written to epoch: 6
root@ssc-vm-g4-rhev4-0786 ~ > hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x99 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0xe0 'default': 'the pool' None None
Services:
    ssc-vm-g4-rhev4-0786.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:10.230.250.52@22001
    [started]  confd               0x7200000000000001:0xa          inet:tcp:10.230.250.52@21001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:10.230.250.52@21002
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:10.230.250.52@21003
    [unknown]  m0_client_other     0x7200000000000001:0x2b         inet:tcp:10.230.250.52@21501
    [offline]  m0_client_other     0x7200000000000001:0x2e         inet:tcp:10.230.250.52@21502
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x35         inet:tcp:10.230.247.111@22001
    [started]  confd               0x7200000000000001:0x38         inet:tcp:10.230.247.111@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:10.230.247.111@21002
    [started]  ioservice           0x7200000000000001:0x4a         inet:tcp:10.230.247.111@21003
    [unknown]  m0_client_other     0x7200000000000001:0x59         inet:tcp:10.230.247.111@21501
    [unknown]  m0_client_other     0x7200000000000001:0x5c         inet:tcp:10.230.247.111@21502
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x63         inet:tcp:10.230.240.142@22001
    [started]  confd               0x7200000000000001:0x66         inet:tcp:10.230.240.142@21001
    [started]  ioservice           0x7200000000000001:0x69         inet:tcp:10.230.240.142@21002
    [started]  ioservice           0x7200000000000001:0x78         inet:tcp:10.230.240.142@21003
    [unknown]  m0_client_other     0x7200000000000001:0x87         inet:tcp:10.230.240.142@21501
    [unknown]  m0_client_other     0x7200000000000001:0x8a         inet:tcp:10.230.240.142@21502
Devices:
    ssc-vm-g4-rhev4-0786.colo.seagate.com
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    ssc-vm-g4-rhev4-0787.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
    ssc-vm-g4-rhev4-0788.colo.seagate.com
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdb
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdi
    [online]  /dev/sdf
root@ssc-vm-g4-rhev4-0786 ~ >
root@ssc-vm-g4-rhev4-0786 ~ > m0client -l 'inet:tcp:10.230.250.52@21502' -H 'inet:tcp:10.230.250.52@22001' -p '<0x7000000000000001:0xe0>' -P '<0x7200000000000001:0x2e>'
m0client >>read 1048680 /tmp/read_1048680_after_rebalanced 4096 200 50
m0client >>quit
Help:
touch   OBJ_ID
write   OBJ_ID SRC_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO UPDATE_FLAG OFFSET
read    OBJ_ID DEST_FILE BLOCK_SIZE BLOCK_COUNT BLOCKS_PER_IO
delete  OBJ_ID
help
quit
root@ssc-vm-g4-rhev4-0786 ~ >
root@ssc-vm-g4-rhev4-0786 ~ > ls -l /tmp/read_1048680_after_repaired
-rw-r--r-- 1 root root 819200 Mar 21 15:44 /tmp/read_1048680_after_repaired
root@ssc-vm-g4-rhev4-0786 ~ >
nkommuri commented 2 years ago

SNS Repair is working with S3 I/O on my 3-node non-k8s cluster. Tested I/O(both read and write) after...

1)Marking a device as failed

2)Marking a device as repairing

3)sns repair

4)marking a device as rebalancing

5)sns rebalance

Everything worked.

https://jts.seagate.com/browse/CORTX-29333?focusedCommentId=2325586&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-2325586

mssawant commented 2 years ago

15N deployment completed with build https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/5411, although Jenkins Job (https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/1219) failed due to HA pod timeout,

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
consul-client-67bh2                                  1/1     Running   0          16m
consul-client-8lhvt                                  1/1     Running   0          18m
consul-client-9d57s                                  1/1     Running   0          17m
consul-client-9xnpj                                  1/1     Running   0          17m
consul-client-jnmsb                                  1/1     Running   0          17m
consul-client-ltsfg                                  1/1     Running   0          17m
consul-client-mmdmw                                  1/1     Running   0          17m
consul-client-npx4b                                  1/1     Running   0          18m
consul-client-rhdw5                                  1/1     Running   0          17m
consul-client-s4b6r                                  1/1     Running   0          16m
consul-client-st9rd                                  1/1     Running   0          18m
consul-client-v2hjb                                  1/1     Running   0          17m
consul-client-v5dqk                                  1/1     Running   0          17m
consul-client-x9ccn                                  1/1     Running   0          16m
consul-client-z775c                                  1/1     Running   0          17m
consul-server-0                                      1/1     Running   0          17m
consul-server-1                                      1/1     Running   0          17m
consul-server-2                                      1/1     Running   0          18m
cortx-control-6d77744bc6-mk9hg                       1/1     Running   0          15m
cortx-data-ssc-vm-g2-rhev4-1630-6877bf8c5f-dzp2v     4/4     Running   0          14m
cortx-data-ssc-vm-g2-rhev4-1631-768cf6974d-j5jvn     4/4     Running   0          14m
cortx-data-ssc-vm-g2-rhev4-1632-5898887f49-7wm69     4/4     Running   0          14m
cortx-data-ssc-vm-g2-rhev4-1635-747f69d4b4-ghkhw     4/4     Running   0          14m
cortx-data-ssc-vm-g2-rhev4-2237-7cdd69d777-5rsrz     4/4     Running   0          14m
cortx-data-ssc-vm-g2-rhev4-2238-69fdddc744-wcgsh     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2107-7d9d974ff4-s2htc     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2184-565bf4ccf5-8fn4j     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2198-568874b7b6-fk97f     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2278-7c49c78457-txzk9     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2279-85fc46bd56-gmfnt     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2281-c5578f7d7-mfdfr      4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2282-78f7b578b-nlpkw      4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2283-7d6b98b597-k5dvz     4/4     Running   0          14m
cortx-data-ssc-vm-g3-rhev4-2284-8566cc444d-m2xgk     4/4     Running   0          14m
cortx-ha-67b498cb6b-2dvc2                            3/3     Running   0          5m11s
cortx-server-ssc-vm-g2-rhev4-1630-c4dc59b47-wvmr4    2/2     Running   0          10m
cortx-server-ssc-vm-g2-rhev4-1631-656d95b755-h7lr9   2/2     Running   0          10m
cortx-server-ssc-vm-g2-rhev4-1632-596bdb6d6c-qpfsp   2/2     Running   0          10m
cortx-server-ssc-vm-g2-rhev4-1635-5447865b85-zvj2l   2/2     Running   0          10m
cortx-server-ssc-vm-g2-rhev4-2237-86b758cd49-g2vrk   2/2     Running   0          10m
cortx-server-ssc-vm-g2-rhev4-2238-64f8b66f59-xpf94   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2107-5bcf9d7c79-fthmm   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2184-58d4955f9c-gh5jq   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2198-9b4749bbc-mkvlx    2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2278-57b46bfd4f-5l69t   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2279-58f76d9cb5-dhlsm   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2281-7bd7cd7d59-vn8vn   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2282-774bddb9c7-6pfj9   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2283-55cc966bdc-7p444   2/2     Running   0          10m
cortx-server-ssc-vm-g3-rhev4-2284-75d5b94465-lnsds   2/2     Running   0          10m
kafka-0                                              1/1     Running   0          16m
kafka-1                                              1/1     Running   0          16m
kafka-2                                              1/1     Running   0          16m
openldap-0                                           1/1     Running   0          18m
openldap-1                                           1/1     Running   0          18m
openldap-2                                           1/1     Running   0          18m
zookeeper-0                                          1/1     Running   0          17m
zookeeper-1                                          1/1     Running   0          17m
zookeeper-2                                          1/1     Running   0          17m
[root@ssc-vm-g2-rhev4-1630 ~]# kubectl exec -it cortx-data-ssc-vm-g2-rhev4-1630-6877bf8c5f-dzp2v --container cortx-hax -- /bin/bash
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]# hctl status -d
Byte_count:
    critical_byte_count : 0
    damaged_byte_count : 0
    degraded_byte_count : 0
    healthy_byte_count : 0
Data pool:
    # fid name
    0x6f00000000000001:0x2d3 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x3f8 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x127        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  ioservice           0x7200000000000001:0x12a        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21001
    [started]  ioservice           0x7200000000000001:0x137        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21002
    [started]  confd               0x7200000000000001:0x144        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1635@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x14b        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  ioservice           0x7200000000000001:0x14e        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21001
    [started]  ioservice           0x7200000000000001:0x15b        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21002
    [started]  confd               0x7200000000000001:0x168        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2107@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x16f        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  ioservice           0x7200000000000001:0x172        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21001
    [started]  ioservice           0x7200000000000001:0x17f        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21002
    [started]  confd               0x7200000000000001:0x18c        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2278@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x193        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  ioservice           0x7200000000000001:0x196        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21001
    [started]  ioservice           0x7200000000000001:0x1a3        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21002
    [started]  confd               0x7200000000000001:0x1b0        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2281@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x1b7        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  ioservice           0x7200000000000001:0x1ba        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21001
    [started]  ioservice           0x7200000000000001:0x1c7        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21002
    [started]  confd               0x7200000000000001:0x1d4        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2184@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x1db        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  ioservice           0x7200000000000001:0x1de        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21001
    [started]  ioservice           0x7200000000000001:0x1eb        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21002
    [started]  confd               0x7200000000000001:0x1f8        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1631@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1630
    [started]  hax                 0x7200000000000001:0x1ff        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@22001
    [started]  ioservice           0x7200000000000001:0x202        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21001
    [started]  ioservice           0x7200000000000001:0x20f        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21002
    [started]  confd               0x7200000000000001:0x21c        inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1630@21003
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1630
    [started]  hax                 0x7200000000000001:0x221        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1630@22001
    [started]  rgw                 0x7200000000000001:0x224        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1630@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    [started]  hax                 0x7200000000000001:0x229        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@22001
    [started]  rgw                 0x7200000000000001:0x22c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2184@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x231        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  rgw                 0x7200000000000001:0x234        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2282@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    [started]  hax                 0x7200000000000001:0x239        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@22001
    [started]  rgw                 0x7200000000000001:0x23c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2281@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x241        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  rgw                 0x7200000000000001:0x244        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2198@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x249        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  rgw                 0x7200000000000001:0x24c        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1632@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    [started]  hax                 0x7200000000000001:0x251        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@22001
    [started]  rgw                 0x7200000000000001:0x254        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1635@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    [started]  hax                 0x7200000000000001:0x259        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@22001
    [started]  rgw                 0x7200000000000001:0x25c        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-1631@21501
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    [started]  hax                 0x7200000000000001:0x261        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  rgw                 0x7200000000000001:0x264        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2238@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x269        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  rgw                 0x7200000000000001:0x26c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2283@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    [started]  hax                 0x7200000000000001:0x271        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@22001
    [started]  rgw                 0x7200000000000001:0x274        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2107@21501
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [started]  hax                 0x7200000000000001:0x2b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@22001
    [started]  ioservice           0x7200000000000001:0x2e         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21001
    [started]  ioservice           0x7200000000000001:0x3b         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21002
    [started]  confd               0x7200000000000001:0x48         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-1632@21003
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0x279        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  rgw                 0x7200000000000001:0x27c        inet:tcp:cortx-server-headless-svc-ssc-vm-g2-rhev4-2237@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0x281        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  rgw                 0x7200000000000001:0x284        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2279@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x289        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  rgw                 0x7200000000000001:0x28c        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2284@21501
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    [started]  hax                 0x7200000000000001:0x291        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@22001
    [started]  rgw                 0x7200000000000001:0x294        inet:tcp:cortx-server-headless-svc-ssc-vm-g3-rhev4-2278@21501
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [started]  hax                 0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@22001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21001
    [started]  ioservice           0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21002
    [started]  confd               0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2282@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238  (RC)
    [started]  hax                 0x7200000000000001:0x4f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@22001
    [started]  ioservice           0x7200000000000001:0x52         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21001
    [started]  ioservice           0x7200000000000001:0x5f         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21002
    [started]  confd               0x7200000000000001:0x6c         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2238@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [started]  hax                 0x7200000000000001:0x73         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@22001
    [started]  ioservice           0x7200000000000001:0x76         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21001
    [started]  ioservice           0x7200000000000001:0x83         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21002
    [started]  confd               0x7200000000000001:0x90         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2283@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [started]  hax                 0x7200000000000001:0x97         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@22001
    [started]  ioservice           0x7200000000000001:0x9a         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21001
    [started]  ioservice           0x7200000000000001:0xa7         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21002
    [started]  confd               0x7200000000000001:0xb4         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2284@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [started]  hax                 0x7200000000000001:0xbb         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@22001
    [started]  ioservice           0x7200000000000001:0xbe         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21001
    [started]  ioservice           0x7200000000000001:0xcb         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21002
    [started]  confd               0x7200000000000001:0xd8         inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2279@21003
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [started]  hax                 0x7200000000000001:0xdf         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@22001
    [started]  ioservice           0x7200000000000001:0xe2         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21001
    [started]  ioservice           0x7200000000000001:0xef         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21002
    [started]  confd               0x7200000000000001:0xfc         inet:tcp:cortx-data-headless-svc-ssc-vm-g2-rhev4-2237@21003
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [started]  hax                 0x7200000000000001:0x103        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@22001
    [started]  ioservice           0x7200000000000001:0x106        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21001
    [started]  ioservice           0x7200000000000001:0x113        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21002
    [started]  confd               0x7200000000000001:0x120        inet:tcp:cortx-data-headless-svc-ssc-vm-g3-rhev4-2198@21003
Devices:
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1635
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2107
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2278
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2281
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2184
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1631
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1630
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1630
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2184
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2282
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2281
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2198
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1632
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1635
    cortx-server-headless-svc-ssc-vm-g2-rhev4-1631
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2238
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2283
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2107
    cortx-data-headless-svc-ssc-vm-g2-rhev4-1632
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g2-rhev4-2237
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2279
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2284
    cortx-server-headless-svc-ssc-vm-g3-rhev4-2278
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2282
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    [online]  /dev/sdc
    [online]  /dev/sdd
    [online]  /dev/sde
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2238
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2283
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2284
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2279
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g2-rhev4-2237
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g3-rhev4-2198
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
[root@cortx-data-headless-svc-ssc-vm-g2-rhev4-1630 /]#