CORTX-31563: Separate ConsulUtil functionality into ConfigManager

[tshaffe1]

I moved the ConsulUtil class out of hax/hax/util.py and into a new source file hax/hax/config.py. I also took some of the Consul-related helpers out of util.py. The class is now named ConfigManager. I updated references elsewhere in the code to use the new name.

This is unfortunately a pretty large patch, since ConsulUtil is/was used throughout the codebase. I tried to keep things to simple renames. I went through and changed type names, as well as variable names where it made sense (e.g. changing local variables named consul_util).

[mssawant]

@trshaffer, great turn around time. Consider following points,

• we can still keep most of the utility/helper functions in utils.py
• but move Consul specific functionality in a separate module, Consul.
• config manager is more of an interface layer that implements certain functions, e.g. kv_get, kv_put, kv_transaction_put, etc.
• Consul class inherits ConfigManager and implements the respective interfaces
• every other module takes ConfigManger as parameter, including utils class.
• all the consul specific implementations are hidden in Consul class.
• This will help us to replace Consul in future with any other ConfigManger implementation by just implementing the respective specific functions.

[mssawant]

@trshaffer, great turn around time. Consider following points,

• we can still keep most of the utility/helper functions in utils.py
• but move Consul specific functionality in a separate module, Consul.
• config manager is more of an interface layer that implements certain functions, e.g. kv_get, kv_put, kv_transaction_put, etc.
• Consul class inherits ConfigManager and implements the respective interfaces
• every other module takes ConfigManger as parameter, including utils class.
• all the consul specific implementations are hidden in Consul class.
• This will help us to replace Consul in future with any other ConfigManger implementation by just implementing the respective specific functions.

[stale[bot]]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 6 days. Tagging @mssawant for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[tshaffe1]

Just pushed changes in response to comments from @mssawant. I set up an abstract base class (ConfigManager) that provides the majority of the functionality (parsing and manipulating fields, using abstract interface). Consul-specific pieces (KV implementation, checking node status, etc.) are exposed as abstract methods. The handful of functions that directly interact with Consul live in the ConsulConfigManager subclass. Type annotations in the codebase have been updated to refer to ConfigManager, and any places that instantiated ConsulUtil now call get_config_manager() which should allow for swapping out the implementation in the future (and may allow for caching/reusing the ConsulConfigManager instance).

Waiting for the deployment tests to run to make sure the refactoring is transparent to the rest of the system

[mssawant]

@trshaffer, please check the deployment and other failues.

[tshaffe1]

@mssawant rebased on the current main. The test failures were due to a bit of code that was using getattr() to compute functions to call, which I missed during the refactor. All tests are green now.

[tshaffe1]

Also note, there are several places in the codebase that directly manipulate the KV store. Those places have type annotations of ConsulConfigManager rather than ConfigManager. It might be good as a future refactor to pull that functionality into the new classes

[vaibhavparatwar]

@supriyachavan4398 lets get this PR tested for 9N or 15N deployment and some existing features for regression. @pavankrishnat can we run regression test suite from CFT team on this build?

[pavankrishnat]

yes @vaibhavparatwar we use https://eos-jenkins.colo.seagate.com/view/QA/job/QA-Sanity-Multinode-RGW/

[supriyachavan4398]

With rgw-integration's branch: "br/sachit/temp-changes-for-hare" created custom build at https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7121/ Did 27N deployment at https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/8613/

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS    RESTARTS        AGE
cortx-consul-client-299l6                            1/1     Running   0               69m
cortx-consul-client-2b9k8                            1/1     Running   0               69m
cortx-consul-client-442vp                            1/1     Running   0               69m
cortx-consul-client-47ftm                            1/1     Running   0               70m
cortx-consul-client-58wkm                            1/1     Running   0               69m
cortx-consul-client-66z99                            1/1     Running   0               70m
cortx-consul-client-88bzq                            1/1     Running   0               71m
cortx-consul-client-89gbf                            1/1     Running   0               69m
cortx-consul-client-9z7cz                            1/1     Running   0               71m
cortx-consul-client-bs5lv                            1/1     Running   0               69m
cortx-consul-client-f9bc7                            1/1     Running   0               69m
cortx-consul-client-hqqgb                            1/1     Running   0               70m
cortx-consul-client-j7xwd                            1/1     Running   0               69m
cortx-consul-client-jbr5p                            1/1     Running   0               71m
cortx-consul-client-k8jgg                            1/1     Running   0               69m
cortx-consul-client-kl8q6                            1/1     Running   0               71m
cortx-consul-client-mgpvh                            1/1     Running   0               70m
cortx-consul-client-n7sbw                            1/1     Running   0               69m
cortx-consul-client-njtrk                            1/1     Running   0               70m
cortx-consul-client-nmmbb                            1/1     Running   0               69m
cortx-consul-client-nshm9                            1/1     Running   0               69m
cortx-consul-client-qqh9g                            1/1     Running   0               69m
cortx-consul-client-qxbtr                            1/1     Running   0               69m
cortx-consul-client-r6rhp                            1/1     Running   0               69m
cortx-consul-client-rjlnm                            1/1     Running   0               69m
cortx-consul-client-tl6gn                            1/1     Running   0               69m
cortx-consul-client-zm5pn                            1/1     Running   0               69m
cortx-consul-server-0                                1/1     Running   1 (25m ago)     70m
cortx-consul-server-1                                1/1     Running   1 (36m ago)     71m
cortx-consul-server-2                                1/1     Running   2 (24m ago)     71m
cortx-control-696869c8fd-wx8cm                       1/1     Running   0               69m
cortx-data-ssc-vm-g2-rhev4-1631-c7854c7c9-nrt96      4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-1632-57fbfd69db-xdxnr     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-1635-55ff86db9f-dvc2x     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-2237-54f8c6ff8-xtjrg      4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-2238-5fcc9c5dc5-6xb2n     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3031-6496d5b7f-6kqgv      4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3156-5b48f8548c-ld52r     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3157-58899f9b68-thfrr     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3158-54fc6596fc-9279l     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3159-7d77c765c4-cbkpz     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3160-66b8d7b948-pjnmj     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3166-645bbb5594-wqjnk     4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3167-c7f767557-ck9c6      4/4     Running   0               68m
cortx-data-ssc-vm-g2-rhev4-3170-659bdb7c77-57w2l     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2107-7d644cb69f-jzq8z     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2184-6d58869858-vgczt     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2198-6c6cffd75f-9g86m     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2278-9cc6d9f6d-d4gq6      4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2279-595fd5f9c9-rttxh     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2281-78b949fb7b-rz4pd     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2282-66768f586f-7p6dl     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2283-7d489bc5f8-9hslf     4/4     Running   0               68m
cortx-data-ssc-vm-g3-rhev4-2284-7b79b7c655-cvwj2     4/4     Running   0               68m
cortx-data-ssc-vm-g4-rhev4-1583-7cc796dc9-k58zn      4/4     Running   0               68m
cortx-data-ssc-vm-g4-rhev4-1590-7f94fb5fd7-cjcdz     4/4     Running   0               68m
cortx-data-ssc-vm-g4-rhev4-1591-d55c76d6d-rfkzg      4/4     Running   0               68m
cortx-data-ssc-vm-g4-rhev4-1592-79785df785-klrph     4/4     Running   0               68m
cortx-ha-5f8f6c9-smbv9                               3/3     Running   0               46m
cortx-kafka-0                                        1/1     Running   2 (72m ago)     73m
cortx-kafka-1                                        1/1     Running   2 (72m ago)     73m
cortx-kafka-2                                        1/1     Running   2 (72m ago)     73m
cortx-server-ssc-vm-g2-rhev4-1631-5f8fbbd9bf-mmvkx   2/2     Running   3 (33m ago)     61m
cortx-server-ssc-vm-g2-rhev4-1632-6c8dbb8f9f-hvw9t   2/2     Running   0               61m
cortx-server-ssc-vm-g2-rhev4-1635-7df4f86cb5-ndk65   2/2     Running   1 (48m ago)     61m
cortx-server-ssc-vm-g2-rhev4-2237-65d468f4cf-8kllq   2/2     Running   2 (38m ago)     61m
cortx-server-ssc-vm-g2-rhev4-2238-7b6c659b49-7fh6v   2/2     Running   1 (43m ago)     61m
cortx-server-ssc-vm-g2-rhev4-3031-646fd56fcc-xbjrz   2/2     Running   5 (21m ago)     61m
cortx-server-ssc-vm-g2-rhev4-3156-7f774679df-ngxm5   2/2     Running   5 (21m ago)     61m
cortx-server-ssc-vm-g2-rhev4-3157-5d9dfcc99f-bmr6j   2/2     Running   0               61m
cortx-server-ssc-vm-g2-rhev4-3158-5db5577fbf-xgh5w   2/2     Running   3 (33m ago)     60m
cortx-server-ssc-vm-g2-rhev4-3159-6bccf7b5bc-pf8q2   2/2     Running   4 (27m ago)     60m
cortx-server-ssc-vm-g2-rhev4-3160-77b78594c-8bn5b    2/2     Running   0               60m
cortx-server-ssc-vm-g2-rhev4-3166-5d989f99-rvtjm     2/2     Running   2 (37m ago)     60m
cortx-server-ssc-vm-g2-rhev4-3167-6cfc7bd97-9rcw4    2/2     Running   1 (48m ago)     60m
cortx-server-ssc-vm-g2-rhev4-3170-fb4df8b5-9994t     2/2     Running   3 (38m ago)     60m
cortx-server-ssc-vm-g3-rhev4-2107-6485675865-4rrnr   2/2     Running   0               60m
cortx-server-ssc-vm-g3-rhev4-2184-ddd565dbc-8h7jv    2/2     Running   6 (15m ago)     60m
cortx-server-ssc-vm-g3-rhev4-2198-9c8c85797-spp2p    2/2     Running   0               60m
cortx-server-ssc-vm-g3-rhev4-2278-86f8c864c5-977b5   2/2     Running   0               60m
cortx-server-ssc-vm-g3-rhev4-2279-7cf5f85bdf-j67b8   2/2     Running   1 (48m ago)     60m
cortx-server-ssc-vm-g3-rhev4-2281-687bb6ffb7-qcgjv   2/2     Running   3 (31m ago)     60m
cortx-server-ssc-vm-g3-rhev4-2282-5754cf8769-q68bs   2/2     Running   7 (7m14s ago)   60m
cortx-server-ssc-vm-g3-rhev4-2283-679788b89f-zpst2   2/2     Running   7 (7m25s ago)   60m
cortx-server-ssc-vm-g3-rhev4-2284-6f88c754d9-csjm2   2/2     Running   4 (27m ago)     60m
cortx-server-ssc-vm-g4-rhev4-1583-db594f7b9-v5hfn    2/2     Running   7 (6m5s ago)    60m
cortx-server-ssc-vm-g4-rhev4-1590-7fcb4ddb5c-gznk9   2/2     Running   0               60m
cortx-server-ssc-vm-g4-rhev4-1591-7877cdcd7f-fhslg   2/2     Running   1 (42m ago)     60m
cortx-server-ssc-vm-g4-rhev4-1592-6597f58dd5-mb4mv   2/2     Running   1 (43m ago)     60m
cortx-zookeeper-0                                    1/1     Running   0               73m
cortx-zookeeper-1                                    1/1     Running   0               73m
cortx-zookeeper-2                                    1/1     Running   0               73m

Seen server pods get restarted. This is a known issue. Happy path IOs:

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 mb s3://test --endpoint-url http://$IP:$Port
make_bucket: test
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-07 05:18:35 test
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# echo "created file..."
created file...
[root@ssc-vm-g2-rhev4-1630 ~]# dd if=/dev/zero of=file_1gb bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.25526 s, 835 MB/s
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Write file..."
Write file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Read file..."
Read file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-07 05:18:55 1048576000 file_1gb
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp s3://test/file_1gb file_1gbDn --endpoint-url http://$IP:$Port
download: s3://test/file_1gb to ./file_1gbDn

cc. @trshaffer, @mssawant, @vaibhavparatwar

[supriyachavan4398]

With rgw-integration's branch: "br/sachit/temp-changes-for-hare" created custom build at https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7121/ Deployed one more 11N (1N Master + 10N Worker) setup at https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/8620/

[root@ssc-vm-g4-rhev4-0710 ~]# kubectl get pods
NAME                                                 READY   STATUS    RESTARTS   AGE
cortx-consul-client-2hncz                            1/1     Running   0          18m
cortx-consul-client-2rpv4                            1/1     Running   0          18m
cortx-consul-client-5dtcv                            1/1     Running   0          17m
cortx-consul-client-8wsf6                            1/1     Running   0          18m
cortx-consul-client-9l9fz                            1/1     Running   0          17m
cortx-consul-client-g7qv5                            1/1     Running   0          18m
cortx-consul-client-h28gl                            1/1     Running   0          18m
cortx-consul-client-k7dzr                            1/1     Running   0          17m
cortx-consul-client-nd8vz                            1/1     Running   0          17m
cortx-consul-client-qr2r6                            1/1     Running   0          18m
cortx-consul-server-0                                1/1     Running   0          17m
cortx-consul-server-1                                1/1     Running   0          18m
cortx-consul-server-2                                1/1     Running   0          18m
cortx-control-7879dcb776-dz6gd                       1/1     Running   0          16m
cortx-data-ssc-vm-g4-rhev4-0711-697b88748f-swwjj     4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-0712-6b648c5d4-bswp8      4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-1587-56b485685-lnpv2      4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-1588-58b889d584-8dpl4     4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-1589-7cbd7576d7-ct5r2     4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-1765-67c4dcc45f-xm8tj     4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-1766-7dcbc5c894-zzchf     4/4     Running   0          15m
cortx-data-ssc-vm-g4-rhev4-1767-5667f64bcc-bhfd8     4/4     Running   0          15m
cortx-data-ssc-vm-rhev4-2450-765d46b87f-drhxc        4/4     Running   0          15m
cortx-data-ssc-vm-rhev4-2451-7d7544648-dw88p         4/4     Running   0          15m
cortx-ha-5cdf5b4bff-2lfjt                            3/3     Running   0          10m
cortx-kafka-0                                        1/1     Running   0          19m
cortx-kafka-1                                        1/1     Running   0          19m
cortx-kafka-2                                        1/1     Running   0          19m
cortx-server-ssc-vm-g4-rhev4-0711-68794df697-djhml   2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-0712-57c9c5bbd7-z24vb   2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-1587-584bb8d877-xt2n7   2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-1588-7bfc5f4bd7-nqtds   2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-1589-759fbcb857-t7r5v   2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-1765-8d956cf85-78sr4    2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-1766-79fb56486c-w2499   2/2     Running   0          13m
cortx-server-ssc-vm-g4-rhev4-1767-7dfb6c7b97-qfx4l   2/2     Running   0          13m
cortx-server-ssc-vm-rhev4-2450-558bcdf69c-xmfg7      2/2     Running   0          13m
cortx-server-ssc-vm-rhev4-2451-685656f7c9-8bthw      2/2     Running   0          13m
cortx-zookeeper-0                                    1/1     Running   0          19m
cortx-zookeeper-1                                    1/1     Running   0          19m
cortx-zookeeper-2                                    1/1     Running   0          19m
[root@ssc-vm-g4-rhev4-0710 ~]# ./exec cortx-data-ssc-vm-g4-rhev4-0711-697b88748f-swwjj
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0711 /]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1588  (RC)
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@21001
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@21002
    [started]  confd               0x7200000000000001:0x3          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1588@22002
    cortx-data-headless-svc-ssc-vm-rhev4-2450
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@22001
    [started]  ioservice           0x7200000000000001:0x5          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@21001
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@21002
    [started]  confd               0x7200000000000001:0x7          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2450@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1588
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1588@22001
    [started]  rgw_s3              0x7200000000000001:0x35         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1588@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1587
    [started]  hax                 0x7200000000000001:0x36         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1587@22001
    [started]  rgw_s3              0x7200000000000001:0x37         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1587@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  rgw_s3              0x7200000000000001:0x39         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1767@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2451
    [started]  hax                 0x7200000000000001:0x3a         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2451@22001
    [started]  rgw_s3              0x7200000000000001:0x3b         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2451@21001
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  ioservice           0x7200000000000001:0x9          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@21002
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1766@22002
    cortx-data-headless-svc-ssc-vm-rhev4-2451
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@21001
    [started]  ioservice           0x7200000000000001:0xe          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@21002
    [started]  confd               0x7200000000000001:0xf          inet:tcp:cortx-data-headless-svc-ssc-vm-rhev4-2451@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1589
    [started]  hax                 0x7200000000000001:0x10         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@22001
    [started]  ioservice           0x7200000000000001:0x11         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@21001
    [started]  ioservice           0x7200000000000001:0x12         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@21002
    [started]  confd               0x7200000000000001:0x13         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1589@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [started]  hax                 0x7200000000000001:0x14         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22001
    [started]  ioservice           0x7200000000000001:0x15         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@21002
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1767@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21001
    [started]  ioservice           0x7200000000000001:0x1a         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@21002
    [started]  confd               0x7200000000000001:0x1b         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0711@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x1c         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  ioservice           0x7200000000000001:0x1d         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21001
    [started]  ioservice           0x7200000000000001:0x1e         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@21002
    [started]  confd               0x7200000000000001:0x1f         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1765@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1587
    [started]  hax                 0x7200000000000001:0x20         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@22001
    [started]  ioservice           0x7200000000000001:0x21         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@21001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@21002
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-1587@22002
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  ioservice           0x7200000000000001:0x25         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21001
    [started]  ioservice           0x7200000000000001:0x26         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@21002
    [started]  confd               0x7200000000000001:0x27         inet:tcp:cortx-data-headless-svc-ssc-vm-g4-rhev4-0712@22002
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1589
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1589@22001
    [started]  rgw_s3              0x7200000000000001:0x29         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1589@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    [started]  hax                 0x7200000000000001:0x2a         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@22001
    [started]  rgw_s3              0x7200000000000001:0x2b         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0711@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@22001
    [started]  rgw_s3              0x7200000000000001:0x2d         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1765@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    [started]  hax                 0x7200000000000001:0x2e         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@22001
    [started]  rgw_s3              0x7200000000000001:0x2f         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-1766@21001
    cortx-server-headless-svc-ssc-vm-rhev4-2450
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2450@22001
    [started]  rgw_s3              0x7200000000000001:0x31         inet:tcp:cortx-server-headless-svc-ssc-vm-rhev4-2450@21001
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712
    [started]  hax                 0x7200000000000001:0x32         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@22001
    [started]  rgw_s3              0x7200000000000001:0x33         inet:tcp:cortx-server-headless-svc-ssc-vm-g4-rhev4-0712@21001
Devices:
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1588
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdf
    [online]  /dev/sdg
    [online]  /dev/sdh
    cortx-data-headless-svc-ssc-vm-rhev4-2450
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1588
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1587
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1767
    cortx-server-headless-svc-ssc-vm-rhev4-2451
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1766
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-rhev4-2451
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1589
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1767
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0711
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1765
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-1587
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-data-headless-svc-ssc-vm-g4-rhev4-0712
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/sdc
    [online]  /dev/sdg
    [online]  /dev/sdh
    [online]  /dev/sdf
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1589
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0711
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1765
    cortx-server-headless-svc-ssc-vm-g4-rhev4-1766
    cortx-server-headless-svc-ssc-vm-rhev4-2450
    cortx-server-headless-svc-ssc-vm-g4-rhev4-0712

Happy path IOs:

[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 mb s3://test --endpoint-url http://$IP:$Port
make_bucket: test
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-07 05:25:31 test
[root@ssc-vm-g4-rhev4-0710 ~]# echo "Write file..."
Write file...
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gb s3://test/file_1gb --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# echo "Read file..."
Read file...
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-07 05:25:53 1048576000 file_1gb
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp s3://test/file_1gb file_1gbDn --endpoint-url http://$IP:$Port
download: s3://test/file_1gb to ./file_1gbDn
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0711 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 0,
    "healthy": 1101004798
  }
}

Degraded IOs:

[root@ssc-vm-g4-rhev4-0710 ~]# kubectl scale deploy cortx-data-ssc-vm-rhev4-2450 --replicas 0
deployment.apps/cortx-data-ssc-vm-rhev4-2450 scaled
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp file_1gb s3://test/file_1gb1 --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb1
[root@cortx-data-headless-svc-ssc-vm-g4-rhev4-0711 /]# hctl status --bytecount
{
  "bytecount": {
    "critical": 0,
    "damaged": 0,
    "degraded": 1105408814,
    "healthy": 1101004798
  }
}
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 ls s3://test/ --endpoint-url http://$IP:$Port
2022-07-07 05:25:53 1048576000 file_1gb
2022-07-07 05:35:10 1048576000 file_1gb1
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp s3://test/file_1gb file_1gbDn --endpoint-url http://$IP:$Port
download: s3://test/file_1gb to ./file_1gbDn
[root@ssc-vm-g4-rhev4-0710 ~]# aws s3 cp s3://test/file_1gb1 file_1gbDn1 --endpoint-url http://$IP:$Port
download: s3://test/file_1gb1 to ./file_1gbDn1

cc. @vaibhavparatwar, @trshaffer, @mssawant

[supriyachavan4398]

Created custom build at https://eos-jenkins.colo.seagate.com/job/GitHub-custom-ci-builds/job/generic/job/custom-ci/7181/console Did 28N deployment at https://eos-jenkins.colo.seagate.com/job/Cortx-kubernetes/job/setup-cortx-cluster-solution-input/607/ with config sns: 18+9+0 and dix: 1+26+0

[root@ssc-vm-g2-rhev4-1630 ~]# kubectl get pods
NAME                                                 READY   STATUS     RESTARTS      AGE
cortx-consul-client-2wxk9                            1/1     Running    0             15m
cortx-consul-client-4bzpv                            1/1     Running    0             16m
cortx-consul-client-4d7cx                            1/1     Running    0             16m
cortx-consul-client-4t8dx                            1/1     Running    0             16m
cortx-consul-client-5wzn7                            1/1     Running    0             16m
cortx-consul-client-6r8hx                            1/1     Running    0             15m
cortx-consul-client-8cbrl                            1/1     Running    0             16m
cortx-consul-client-92cg9                            1/1     Running    0             15m
cortx-consul-client-94flm                            1/1     Running    0             16m
cortx-consul-client-9l552                            1/1     Running    0             16m
cortx-consul-client-9rspc                            1/1     Running    0             16m
cortx-consul-client-bhbv7                            1/1     Running    0             16m
cortx-consul-client-df4m9                            1/1     Running    0             16m
cortx-consul-client-f7b2p                            1/1     Running    0             17m
cortx-consul-client-ggthl                            1/1     Running    0             16m
cortx-consul-client-j24g7                            1/1     Running    0             15m
cortx-consul-client-k76l7                            1/1     Running    0             15m
cortx-consul-client-lmlg6                            1/1     Running    0             16m
cortx-consul-client-mdcbh                            1/1     Running    0             15m
cortx-consul-client-mq5cl                            1/1     Running    0             16m
cortx-consul-client-nhsjz                            1/1     Running    0             16m
cortx-consul-client-qwcgp                            1/1     Running    0             15m
cortx-consul-client-s6plb                            1/1     Running    0             15m
cortx-consul-client-v2qh4                            1/1     Running    0             15m
cortx-consul-client-vr6rl                            1/1     Running    0             15m
cortx-consul-client-wd9k2                            1/1     Running    0             16m
cortx-consul-client-whsl9                            1/1     Running    0             15m
cortx-consul-server-0                                1/1     Running    0             15m
cortx-consul-server-1                                1/1     Running    0             16m
cortx-consul-server-2                                1/1     Running    0             17m
cortx-control-74589fcd7b-4n52b                       1/1     Running    0             14m
cortx-data-ssc-vm-g2-rhev4-1631-65f465cc5-pb42m      4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-1632-5db98f7b58-6pwxq     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-1635-74b6f5866c-s5xj8     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-2237-6f7c54459d-qmx5g     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-2238-797bb7695-vl5cl      4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3031-74cd647975-9nw5d     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3156-7655575f47-ntkn5     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3157-f9fff7c64-hjrfm      4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3158-54f89df44f-m76cd     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3159-996545d46-cswmr      4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3160-5695d98644-8mhht     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3166-84f9bc4c54-hjpq9     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3167-56d66c7b89-pd77s     4/4     Running    0             13m
cortx-data-ssc-vm-g2-rhev4-3170-7946f97c45-cwfnr     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2107-59b974684c-dzgbz     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2184-5775dc8466-2fr4j     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2198-7f59894975-bwm6q     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2278-5c8c6485dd-52lsw     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2279-85dbf4c987-m4qzj     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2281-686c7dcf9b-z6hjf     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2282-6489597749-8sfl8     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2283-6c88d8447d-445x6     4/4     Running    0             13m
cortx-data-ssc-vm-g3-rhev4-2284-7c4b6667f5-6h42j     4/4     Running    0             13m
cortx-data-ssc-vm-g4-rhev4-1583-7c484db875-pdwk7     4/4     Running    0             13m
cortx-data-ssc-vm-g4-rhev4-1590-6855bfd755-4gls5     4/4     Running    0             13m
cortx-data-ssc-vm-g4-rhev4-1591-78c96bdd88-wg8rk     4/4     Running    0             13m
cortx-data-ssc-vm-g4-rhev4-1592-5bf546d97c-mhhhg     4/4     Running    0             13m
cortx-ha-68d4fcbcdc-jm4qv                            0/3     Init:0/1   0             115s
cortx-kafka-0                                        1/1     Running    0             18m
cortx-kafka-1                                        1/1     Running    0             18m
cortx-kafka-2                                        1/1     Running    1 (17m ago)   18m
cortx-server-ssc-vm-g2-rhev4-1631-5c97cdf47-qq85l    2/2     Running    0             8m50s
cortx-server-ssc-vm-g2-rhev4-1632-69bb8c867c-w678s   2/2     Running    0             8m49s
cortx-server-ssc-vm-g2-rhev4-1635-64dbd76db7-s94jn   2/2     Running    0             8m49s
cortx-server-ssc-vm-g2-rhev4-2237-66675b7cfc-4hbp5   2/2     Running    0             8m48s
cortx-server-ssc-vm-g2-rhev4-2238-5684c7d995-8k722   2/2     Running    0             8m47s
cortx-server-ssc-vm-g2-rhev4-3031-78f49f6dd7-nwvqw   2/2     Running    0             8m47s
cortx-server-ssc-vm-g2-rhev4-3156-c58ffd55f-2b6bd    2/2     Running    0             8m46s
cortx-server-ssc-vm-g2-rhev4-3157-bbcb89ffc-l75rv    2/2     Running    0             8m45s
cortx-server-ssc-vm-g2-rhev4-3158-7888949987-lc8dz   2/2     Running    0             8m45s
cortx-server-ssc-vm-g2-rhev4-3159-8c7ff77d9-blz4r    2/2     Running    0             8m44s
cortx-server-ssc-vm-g2-rhev4-3160-7dcc956d45-zr7l9   2/2     Running    0             8m44s
cortx-server-ssc-vm-g2-rhev4-3166-6759b44645-cm6pf   2/2     Running    0             8m43s
cortx-server-ssc-vm-g2-rhev4-3167-df9cc5fcf-sxfl6    2/2     Running    0             8m42s
cortx-server-ssc-vm-g2-rhev4-3170-67f7f96879-pj9zh   2/2     Running    0             8m42s
cortx-server-ssc-vm-g3-rhev4-2107-8c7785bb7-l94h7    2/2     Running    0             8m41s
cortx-server-ssc-vm-g3-rhev4-2184-7f89777f4f-z8mjt   2/2     Running    0             8m40s
cortx-server-ssc-vm-g3-rhev4-2198-f9556bb7-jwvtv     2/2     Running    0             8m40s
cortx-server-ssc-vm-g3-rhev4-2278-58bc7f8497-zc77j   2/2     Running    0             8m39s
cortx-server-ssc-vm-g3-rhev4-2279-86679459ff-6pf92   2/2     Running    0             8m38s
cortx-server-ssc-vm-g3-rhev4-2281-66b69c7fcc-x9h9l   2/2     Running    0             8m38s
cortx-server-ssc-vm-g3-rhev4-2282-5956c7c7cf-bc58z   2/2     Running    0             8m37s
cortx-server-ssc-vm-g3-rhev4-2283-7784b65bc7-vz6bf   2/2     Running    0             8m37s
cortx-server-ssc-vm-g3-rhev4-2284-56f7fd54dc-hw6wt   2/2     Running    0             8m36s
cortx-server-ssc-vm-g4-rhev4-1583-7dd46d967-zqwpw    2/2     Running    0             8m35s
cortx-server-ssc-vm-g4-rhev4-1590-686d4b4d9f-dkwz2   2/2     Running    0             8m35s
cortx-server-ssc-vm-g4-rhev4-1591-7b6b899645-htwfd   2/2     Running    0             8m34s
cortx-server-ssc-vm-g4-rhev4-1592-5cbd7f46c5-89xfd   2/2     Running    0             8m33s
cortx-zookeeper-0                                    1/1     Running    0             18m
cortx-zookeeper-1                                    1/1     Running    0             18m
cortx-zookeeper-2                                    1/1     Running    0             18m

Happy path IOs:

[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 mb s3://test --endpoint-url http://$IP:$Port
make_bucket: test
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 ls --endpoint-url http://$IP:$Port
2022-07-12 00:22:41 test
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# echo "created file..."
created file...
[root@ssc-vm-g2-rhev4-1630 ~]# dd if=/dev/zero of=file_1gb bs=10M count=100
100+0 records in
100+0 records out
1048576000 bytes (1.0 GB) copied, 1.6238 s, 646 MB/s
[root@ssc-vm-g2-rhev4-1630 ~]#
[root@ssc-vm-g2-rhev4-1630 ~]# echo "Write file..."
Write file...
[root@ssc-vm-g2-rhev4-1630 ~]# aws s3 cp file_1gb s3://test/file_1gb --endpoint-url http://$IP:$Port
upload: ./file_1gb to s3://test/file_1gb

[vaibhavparatwar]

@mssawant so with these chanages 27N deployment and happy path I/Os are working fine. rest of the testing can be done on 11 or 15N ?

[stale[bot]]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 6 days. Tagging @mssawant for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[stale[bot]]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 6 days. Tagging @mssawant for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[seagate-sarang-sawant]

retest this please

[stale[bot]]

This issue/pull request has been marked as needs attention as it has been left pending without new activity for 6 days. Tagging @mssawant for appropriate assignment. Sorry for the delay & Thank you for contributing to CORTX. We will get back to you as soon as possible.

[shailesh-vaidya]

Closing as an obsolete