search

IBM / ibmcloud-storage-utilities

Helper Storage utilities for configuration of ibm cloud block storage with IBM Container Service (IKS) clusters
Apache License 2.0
5 stars 27 forks source link

Portworx DR feature validation #90

Closed mumutyal closed 4 years ago

mumutyal commented 4 years ago

Prerequisites Kubernetes Clusters: At least two Kubernetes clusters which are part of the same metropolitan area network with a maximum network latency of 10ms between them Version: A single Portworx cluster of v2.1 or later installed on across Kubernetes clusters and Stork v2.2.4 or later on both clusters Network Connectivity: Ports 9001 to 9020 open between the two Kubernetes clusters External Kvdb: A kvdb like etcd or consul setup outside of the Kubernetes clusters License: A DR enabled PX-Enterprise license on both the source and destination clusters to use this feature

The PX DR feature requires 2 IKS clusters . I took two clusters pxdrsync04 pxdrsync03

The nodes of the 2 IKS cluster

Nodes of the cluster 1 kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 10.183.63.38 Ready 18h v1.16.8+IKS 10.183.63.38 169.47.40.209 Ubuntu 16.04.6 LTS 4.4.0-177-generic containerd://1.3.3 10.183.63.49 Ready 18h v1.16.8+IKS 10.183.63.49 169.47.40.210 Ubuntu 16.04.6 LTS 4.4.0-177-generic containerd://1.3.3 10.183.63.56 Ready 18h v1.16.8+IKS 10.183.63.56 169.47.40.214 Ubuntu 16.04.6 LTS 4.4.0-177-generic containerd://1.3.3

Nodes of the cluster 2 kubectl get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 10.183.63.29 Ready 46h v1.16.8+IKS 10.183.63.29 169.47.63.243 Ubuntu 16.04.6 LTS 4.4.0-177-generic containerd://1.3.3 10.183.63.37 Ready 46h v1.16.8+IKS 10.183.63.37 169.47.63.245 Ubuntu 16.04.6 LTS 4.4.0-177-generic containerd://1.3.3 10.183.63.40 Ready 46h v1.16.8+IKS 10.183.63.40 169.47.40.216 Ubuntu 16.04.6 LTS 4.4.0-177-generic containerd://1.3.3 root@jupiter-vm1117:~#

mumutyal commented 4 years ago

In this mode of operation, a single Portworx cluster will stretch across multiple Kubernetes clusters.

A single Portworx cluster running across both the Kubernetes clusters

root@jupiter-vm1117 kubectl exec -it portworx-lcxrv -n kube-system /opt/pwx/bin/pxctl status Status: PX is operational License: PX-Enterprise IBM Cloud DR (test) (expires in 285 days) Node ID: dbca363a-a450-40d5-bcdd-7974e8a37503 IP: 10.183.63.40 Local Storage Pool: 1 pool POOL IO_PRIORITY RAID_LEVEL USABLE USED STATUS ZONE REGION 0 LOW raid0 30 GiB 3.5 GiB Online wdc04 us-east Local Storage Devices: 1 device Device Path Media Type Size Last-Scan 0:1 /dev/mapper/3600a098038304750692b4c78754e6272 STORAGE_MEDIUM_MAGNETIC 30 GiB 17 Apr 20 13:44 UTC total - 30 GiB Cache Devices: No cache devices Cluster Summary Cluster ID: pxdrtest000987654 Cluster UUID: 85f8b0d1-1212-4ce2-8967-6e250fbd2261 Scheduler: kubernetes Nodes: 6 node(s) with storage (6 online) IP ID SchedulerNodeName StorageNode Used Capacity Status StorageStatus Version Kernel OS 10.183.63.37 e72e0e0f-05ca-453b-92e9-adebb703be22 10.183.63.37 Yes 3.5 GiB 30 GiB Online Up 2.5.0.0-5d86e34 4.4.0-177-generic Ubuntu 16.04.6 LTS 10.183.63.29 e6f141a7-7240-44cb-8b15-9df55e3cf8dd 10.183.63.29 Yes 3.5 GiB 30 GiB Online Up 2.5.0.0-5d86e34 4.4.0-177-generic Ubuntu 16.04.6 LTS 10.183.63.49 dd56f038-aab5-4f75-9067-a316699567ee 10.183.63.49 Yes 3.0 GiB 20 GiB Online Up 2.5.0.0-5d86e34 4.4.0-177-generic Ubuntu 16.04.6 LTS 10.183.63.40 dbca363a-a450-40d5-bcdd-7974e8a37503 10.183.63.40 Yes 3.5 GiB 30 GiB Online Up (This node) 2.5.0.0-5d86e34 4.4.0-177-generic Ubuntu 16.04.6 LTS 10.183.63.38 44bcf07b-31a3-4482-ad29-34188565540c 10.183.63.38 Yes 3.0 GiB 20 GiB Online Up 2.5.0.0-5d86e34 4.4.0-177-generic Ubuntu 16.04.6 LTS 10.183.63.56 34600105-614e-4bb3-bd6c-6008fa47729d 10.183.63.56 Yes 3.0 GiB 20 GiB Online Up 2.5.0.0-5d86e34 4.4.0-177-generic Ubuntu 16.04.6 LTS Warnings: WARNING: Persistent journald logging is not enabled on this node. Global Storage Pool Total Used : 20 GiB Total Capacity : 150 GiB root@jupiter-vm1117:~#

mumutyal commented 4 years ago

root@jupiter-vm1117:~# storkctl get clusterdomainsstatus NAME LOCAL-DOMAIN ACTIVE INACTIVE CREATED pxdrtest000987654 [] [] 17 Apr 20 09:45 EDT

mumutyal commented 4 years ago

Got some issue creating the cluster pair checking with portworx team

mumutyal commented 4 years ago

Cluster domain status

root@kube-bqcp815w0v8h776u0g10-pxdrsync04-default-00000227:/# root@jupiter-vm1117:~# storkctl get clusterdomainsstatus NAME LOCAL-DOMAIN ACTIVE INACTIVE CREATED pxdrtest000987654 wdc04-1 [wdc04-1 (InSync), wdc04-2 (InSync)] [] 17 Apr 20 10:49 EDT root@jupiter-vm1117:~#

After the cluster domain deactivate

root@jupiter-vm1117:~/PXDRTest# storkctl deactivate clusterdomain wdc04-2 Cluster Domain deactivate operation started successfully for wdc04-2 root@jupiter-vm1117:~/PXDRTest# storkctl get clusterdomainsstatus NAME LOCAL-DOMAIN ACTIVE INACTIVE CREATED pxdrtest000987654 wdc04-2 [wdc04-1 (InSync)] [wdc04-2 (NotInSync)] 17 Apr 20 09:45 EDT root@jupiter-vm1117:~/PXDRTest#

mumutyal commented 4 years ago

Tested the Async DR feature on portworx created the record and uploaded at the following location https://ibm.ent.box.com/folder/111605638540