search

ocp-power-automation / ocp4-upi-powervm

OpenShift on IBM PowerVM servers managed using PowerVC
Apache License 2.0
27 stars 51 forks source link

OCP 4.x cluster with vSCSI connection type fails to scan and identify the attached disk while setting up NFS #82

Closed arcprabh closed 4 years ago

arcprabh commented 4 years ago

Problem :

NFS disk link creation fails when the vscsi protocol is used for storage connection type for VMs in OCP 4.x cluster deployment.

GH bug reference - https://github.ibm.com/redstack-power/project-mgmt/issues/469

Analysis :

That is because the create_disk_link script is specifically checking only for multipath devices and vscsi attached disks will not be listed with that command.

The below error is seen.

module.install.module.bastion.null_resource.setup_nfs_disk[0]: Provisioning with 'remote-exec'...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Connecting to remote host via SSH...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Host: 9.114.97.20
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   User: root
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Password: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Private key: true
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Certificate: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   SSH Agent: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Checking Host Key: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Connected!
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Syncing file systems
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning SCSI subsystem for new devices and remove devices that have disappeared
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 1 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 01 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VOPTA            Rev:
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   CD-ROM                           ANSI SCSI revision: 04
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 2 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 2 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 02 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VDASD            Rev: 0001
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 03
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 3 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 3 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 03 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VDASD            Rev: 0001
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 03
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 4 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 5 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 6 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 7 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 new or changed device(s) found.          
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 remapped or resized device(s) found.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 device(s) removed.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): ls: cannot access '/dev/mapper/mpath*': No such file or directory
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.bootstrap[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.worker[1]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.worker[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[1]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[2]: Still creating... [10s elapsed]
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): mke2fs 1.45.4 (23-Sep-2019)

module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): The file /dev/disk/pv-storage-disk does not exist and no size was specified.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): mount: /export: special device /dev/disk/pv-storage-disk does not exist.

TF script fails as follows:

module.install.module.install.null_resource.config: Still creating... [4m11s elapsed]
module.install.module.install.null_resource.config (remote-exec): <localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1596540417.9519665-8296-214222424012953/ > /dev/null 2>&1 && sleep 0'
module.install.module.install.null_resource.config (remote-exec): failed: [localhost] (item=nfs-server) => {
module.install.module.install.null_resource.config (remote-exec):     "ansible_loop_var": "item",
module.install.module.install.null_resource.config (remote-exec):     "changed": false,
module.install.module.install.null_resource.config (remote-exec):     "invocation": {
module.install.module.install.null_resource.config (remote-exec):         "module_args": {
module.install.module.install.null_resource.config (remote-exec):             "daemon_reexec": false,
module.install.module.install.null_resource.config (remote-exec):             "daemon_reload": false,
module.install.module.install.null_resource.config (remote-exec):             "enabled": true,
module.install.module.install.null_resource.config (remote-exec):             "force": null,
module.install.module.install.null_resource.config (remote-exec):             "masked": null,
module.install.module.install.null_resource.config (remote-exec):             "name": "nfs-server",
module.install.module.install.null_resource.config (remote-exec):             "no_block": false,
module.install.module.install.null_resource.config (remote-exec):             "scope": null,
module.install.module.install.null_resource.config (remote-exec):             "state": "started",
module.install.module.install.null_resource.config (remote-exec):             "user": null
module.install.module.install.null_resource.config (remote-exec):         }
module.install.module.install.null_resource.config (remote-exec):     },
module.install.module.install.null_resource.config (remote-exec):     "item": "nfs-server",
module.install.module.install.null_resource.config (remote-exec):     "msg": "Unable to start service nfs-server: A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.\n"
module.install.module.install.null_resource.config (remote-exec): }

module.install.module.install.null_resource.config (remote-exec): RUNNING HANDLER [restart bind] *************************************************
module.install.module.install.null_resource.config (remote-exec): task path: /root/ocp4-helpernode/handlers/main.yml:3

module.install.module.install.null_resource.config (remote-exec): RUNNING HANDLER [restart haproxy] **********************************************
module.install.module.install.null_resource.config (remote-exec): task path: /root/ocp4-helpernode/handlers/main.yml:8

module.install.module.install.null_resource.config (remote-exec): RUNNING HANDLER [restart httpd] ************************************************
module.install.module.install.null_resource.config (remote-exec): task path: /root/ocp4-helpernode/handlers/main.yml:13

module.install.module.install.null_resource.config (remote-exec): RUNNING HANDLER [restart dhcpd] ************************************************
module.install.module.install.null_resource.config (remote-exec): task path: /root/ocp4-helpernode/handlers/main.yml:18

module.install.module.install.null_resource.config (remote-exec): RUNNING HANDLER [restart tftp] *************************************************
module.install.module.install.null_resource.config (remote-exec): task path: /root/ocp4-helpernode/handlers/main.yml:23

module.install.module.install.null_resource.config (remote-exec): RUNNING HANDLER [restart nfs] **************************************************
module.install.module.install.null_resource.config (remote-exec): task path: /root/ocp4-helpernode/handlers/main.yml:28

module.install.module.install.null_resource.config (remote-exec): PLAY RECAP *********************************************************************
module.install.module.install.null_resource.config (remote-exec): localhost                  : ok=47   changed=31   unreachable=0    failed=1    skipped=20   rescued=1    ignored=0

Error: error executing "/tmp/terraform_1541468865.sh": Process exited with status 32
Error: error executing "/tmp/terraform_1801239085.sh": Process exited with status 2

The commands used in https://github.com/ocp-power-automation/ocp4-upi-powervm/blob/master/modules/1_bastion/templates/create_disk_link.sh is not suitable for vscsi attached devices, so we have to use another method for identifying newly attached vscsi disks.

The below existing method for identifying disks works well only for NPIV/multipath devices.

ls -1 /dev/mapper/mpath*|egrep -v "[0-9]$"

[root@arc-csi-f350-bastion ~]# lsblk
NAME                      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                         8:0    0   300G  0 disk  
`-mpathf                  253:0    0   300G  0 mpath 
  |-mpathf1               253:2    0     4M  0 part  
  |-mpathf2               253:3    0     1G  0 part  /boot
  `-mpathf3               253:4    0   199G  0 part  
    |-rhel_icp4all73-root 253:5    0 194.9G  0 lvm   /
    `-rhel_icp4all73-swap 253:6    0   4.1G  0 lvm   [SWAP]
sdb                         8:16   0   300G  0 disk  
`-mpathg                  253:1    0   300G  0 mpath /export
sdc                         8:32   0   300G  0 disk  
`-mpathf                  253:0    0   300G  0 mpath 
  |-mpathf1               253:2    0     4M  0 part  
  |-mpathf2               253:3    0     1G  0 part  /boot
  `-mpathf3               253:4    0   199G  0 part  
    |-rhel_icp4all73-root 253:5    0 194.9G  0 lvm   /
    `-rhel_icp4all73-swap 253:6    0   4.1G  0 lvm   [SWAP]
sdd                         8:48   0   300G  0 disk  
`-mpathg                  253:1    0   300G  0 mpath /export
sde                         8:64   0   300G  0 disk  
`-mpathf                  253:0    0   300G  0 mpath 
  |-mpathf1               253:2    0     4M  0 part  
  |-mpathf2               253:3    0     1G  0 part  /boot
  `-mpathf3               253:4    0   199G  0 part  
    |-rhel_icp4all73-root 253:5    0 194.9G  0 lvm   /
    `-rhel_icp4all73-swap 253:6    0   4.1G  0 lvm   [SWAP]
sdf                         8:80   0   300G  0 disk  
`-mpathg                  253:1    0   300G  0 mpath /export
sdg                         8:96   0   300G  0 disk  
`-mpathf                  253:0    0   300G  0 mpath 
  |-mpathf1               253:2    0     4M  0 part  
  |-mpathf2               253:3    0     1G  0 part  /boot
  `-mpathf3               253:4    0   199G  0 part  
    |-rhel_icp4all73-root 253:5    0 194.9G  0 lvm   /
    `-rhel_icp4all73-swap 253:6    0   4.1G  0 lvm   [SWAP]
sdh                         8:112  0   300G  0 disk  
`-mpathg                  253:1    0   300G  0 mpath /export

The below method works well for identifying vscsi attached disks. We need to incorporate this into the NFS disk identification and disk link method.

ls -1 /dev/sd*|egrep -v "[0-9]$"

[root@arc45-vsc-f3bd-bastion ~]# lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                       8:0    0   300G  0 disk 
|-sda1                    8:1    0     4M  0 part 
|-sda2                    8:2    0     1G  0 part /boot
`-sda3                    8:3    0   199G  0 part 
  |-rhel_icp4all73-root 253:0    0 194.9G  0 lvm  /
  `-rhel_icp4all73-swap 253:1    0   4.1G  0 lvm  [SWAP]
sdb                       8:16   0   400G  0 disk

I have included the below changes to handle both NPIV and vSCSI attached disks to the create_disk_link.sh script.

Link to the PR - https://github.com/arcprabh/ocp4-upi-powervm/pull/1

I have tested the OCP 4.5 cluster deployment with vSCSI connection type and NFS storage class and it works fine. Check the below NFS disk attach messages from TF logs.

module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Syncing file systems
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning SCSI subsystem for new devices and remove devices that have disappeared
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 1 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 01 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VOPTA            Rev:
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   CD-ROM                           ANSI SCSI revision: 04
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 2 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 2 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 02 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VDASD            Rev: 0001
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 03
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 3 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 3 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 03 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VDASD            Rev: 0001
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 03
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 4 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 5 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 6 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 7 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 new or changed device(s) found.          
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 remapped or resized device(s) found.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 device(s) removed.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): ls: cannot access '/dev/mapper/mpath*': No such file or directory
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): vSCSI devices detected on helper node.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): ENV{DEVTYPE}=="disk", ENV{SUBSYSTEM}=="block", ENV{DEVPATH}=="/devices/vio/30000002/host0/target0:0:3/0:0:3:0/block/sdb" SYMLINK+="disk/pv-storage-disk"
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[1]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[2]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.bootstrap[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.worker[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.worker[1]: Still creating... [10s elapsed]
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): mke2fs 1.45.4 (23-Sep-2019)
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Creating filesystem with 104857600 4k blocks and 26214400 inodes
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Filesystem UUID: 31520577-de52-4eff-a9a3-2d85feeb2a24
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Superblock backups stored on blocks:
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):    102400000

module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Allocating group tables:    0/3200
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): done     
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Writing inode tables: done     
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Creating journal (262144 blocks):
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): done
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Writing superblocks and filesystem accounting information: done

With this patch, for the first time, I have been able to deploy an OCP 4.5 cluster using vSCSI connection type successfully.

Apply complete! Resources: 32 added, 0 changed, 0 destroyed.

Outputs:

bastion_ip = 9.114.97.33
bastion_ssh_command = ssh root@9.114.97.33
bootstrap_ip = 9.114.97.17
etc_hosts_entries = 

9.114.97.33 api.arc-vscsi-5ed9.redhat.com console-openshift-console.apps.arc-vscsi-5ed9.redhat.com integrated-oauth-server-openshift-authentication.apps.arc-vscsi-5ed9.redhat.com oauth-openshift.apps.arc-vscsi-5ed9.redhat.com prometheus-k8s-openshift-monitoring.apps.arc-vscsi-5ed9.redhat.com grafana-openshift-monitoring.apps.arc-vscsi-5ed9.redhat.com example.apps.arc-vscsi-5ed9.redhat.com

master_ips = [
  "9.114.97.25",
  "9.114.97.22",
  "9.114.97.26",
]
oc_server_url = https://api.arc-vscsi-5ed9.redhat.com:6443/
storageclass_name = nfs-storage-provisioner
web_console_url = https://console-openshift-console.apps.arc-vscsi-5ed9.redhat.com
worker_ips = [
  "9.114.97.29",
  "9.114.97.32",
]

[root@arc-vscsi-5ed9-bastion ~]# oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication                             4.5.4     True        False         False      15m
cloud-credential                           4.5.4     True        False         False      42m
cluster-autoscaler                         4.5.4     True        False         False      26m
config-operator                            4.5.4     True        False         False      26m
console                                    4.5.4     True        False         False      16m
csi-snapshot-controller                    4.5.4     True        False         False      22m
dns                                        4.5.4     True        False         False      35m
etcd                                       4.5.4     True        False         False      33m
image-registry                             4.5.4     True        False         False      13m
ingress                                    4.5.4     True        False         False      22m
insights                                   4.5.4     True        False         False      27m
kube-apiserver                             4.5.4     True        False         False      32m
kube-controller-manager                    4.5.4     True        False         False      34m
kube-scheduler                             4.5.4     True        False         False      34m
kube-storage-version-migrator              4.5.4     True        False         False      23m
machine-api                                4.5.4     True        False         False      27m
machine-approver                           4.5.4     True        False         False      32m
machine-config                             4.5.4     True        False         False      35m
marketplace                                4.5.4     True        False         False      27m
monitoring                                 4.5.4     True        False         False      16m
network                                    4.5.4     True        False         False      37m
node-tuning                                4.5.4     True        False         False      37m
openshift-apiserver                        4.5.4     True        False         False      29m
openshift-controller-manager               4.5.4     True        False         False      12m
openshift-samples                          4.5.4     True        False         False      26m
operator-lifecycle-manager                 4.5.4     True        False         False      36m
operator-lifecycle-manager-catalog         4.5.4     True        False         False      36m
operator-lifecycle-manager-packageserver   4.5.4     True        False         False      30m
service-ca                                 4.5.4     True        False         False      36m
storage                                    4.5.4     True        False         False      27m

[root@arc-vscsi-5ed9-bastion ~]# oc get pods --all-namespaces | grep -vi completed | grep -vi running
NAMESPACE                                          NAME                                                          READY   STATUS      RESTARTS   AGE

[root@arc-vscsi-5ed9-bastion ~]# oc get sc
NAME                                PROVISIONER   RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
nfs-storage-provisioner (default)   nfs-storage   Delete          Immediate           false                  14m

[root@arc-vscsi-5ed9-bastion ~]# oc get pvc --all-namespaces
NAMESPACE                  NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS              AGE
openshift-image-registry   registry-pvc   Bound    pvc-6670d32e-322a-446e-964f-8db5ed3b0a8a   20Gi       RWX            nfs-storage-provisioner   13m

[root@arc-vscsi-5ed9-bastion ~]# lsblk
NAME                    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                       8:0    0   300G  0 disk 
|-sda1                    8:1    0     4M  0 part 
|-sda2                    8:2    0     1G  0 part /boot
`-sda3                    8:3    0   199G  0 part 
  |-rhel_icp4all73-root 253:0    0 194.9G  0 lvm  /
  `-rhel_icp4all73-swap 253:1    0   4.1G  0 lvm  [SWAP]
sdb                       8:16   0   400G  0 disk /export
sr0                      11:0    1   492K  0 rom  

[root@arc-vscsi-5ed9-bastion ~]# oc version
Client Version: 4.5.4
Server Version: 4.5.4
Kubernetes Version: v1.18.3+012b3ec

[root@arc-vscsi-5ed9-bastion ~]# oc get nodes
NAME                                 STATUS   ROLES    AGE   VERSION
master-0.arc-vscsi-5ed9.redhat.com   Ready    master   39m   v1.18.3+b74c5ed
master-1.arc-vscsi-5ed9.redhat.com   Ready    master   39m   v1.18.3+b74c5ed
master-2.arc-vscsi-5ed9.redhat.com   Ready    master   35m   v1.18.3+b74c5ed
worker-0.arc-vscsi-5ed9.redhat.com   Ready    worker   25m   v1.18.3+b74c5ed
worker-1.arc-vscsi-5ed9.redhat.com   Ready    worker   25m   v1.18.3+b74c5ed

With this patch, I have also triggerd the script for creating another OCP 4.5 cluster with NPIV connection to ensure that the else part of the loop is getting invoked where I moved the original code after adding an if conditional check. 

module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Connecting to remote host via SSH...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Host: 9.114.97.19
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   User: root
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Password: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Private key: true
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Certificate: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   SSH Agent: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):   Checking Host Key: false
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Connected!
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Syncing file systems
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning SCSI subsystem for new devices and remove devices that have disappeared
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 1 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi0 Channel: 00 Id: 01 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: AIX      Model: VOPTA            Rev:
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   CD-ROM                           ANSI SCSI revision: 04
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 2 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 3 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.bootstrap[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.worker[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.worker[1]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[0]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[1]: Still creating... [10s elapsed]
module.install.module.nodes.openstack_compute_instance_v2.master[2]: Still creating... [10s elapsed]
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 4 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 5 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 6 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 0 0 7 0 ...           
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):                                            
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning host 1 for  all SCSI target IDs, all LUNs
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 1 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 1 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 1 0 0 1 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi1 Channel: 00 Id: 00 Lun: 01
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 1 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 1 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 1 0 1 1 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi1 Channel: 00 Id: 01 Lun: 01
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Scanning host 2 for  all SCSI target IDs, all LUNs
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 2 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 2 0 0 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 2 0 0 1 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi2 Channel: 00 Id: 00 Lun: 01
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 2 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 2 0 1 0 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi2 Channel: 00 Id: 01 Lun: 00
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):  Scanning for device 2 0 1 1 ...
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): OLD: Host: scsi2 Channel: 00 Id: 01 Lun: 01
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Vendor: IBM      Model: 2145             Rev: 0000
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):       Type:   Direct-Access                    ANSI SCSI revision: 06
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 new or changed device(s) found.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 remapped or resized device(s) found.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): 0 device(s) removed.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): NPIV devices detected on helper node.
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): ENV{DEVTYPE}=="disk", ENV{SUBSYSTEM}=="block", ENV{DEVPATH}=="/devices/virtual/block/dm-1" SYMLINK+="disk/pv-storage-disk"
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Disk not ready, sleeping for 2s..
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): mke2fs 1.45.4 (23-Sep-2019)
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Creating filesystem with 104857600 4k blocks and 26214400 inodes
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Filesystem UUID: 25834b59-6bb9-4270-ab3d-9d42ae2e05a8
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Superblock backups stored on blocks:
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):    102400000

module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Allocating group tables: done     
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Writing inode tables: done     
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Creating journal (262144 blocks):
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): done
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): Writing superblocks and filesystem accounting information:
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec):          
module.install.module.bastion.null_resource.setup_nfs_disk[0] (remote-exec): done

The NPIV deploy script is still running. Will update once done.

arcprabh commented 4 years ago

PR - https://github.com/arcprabh/ocp4-upi-powervm/pull/1

arcprabh commented 4 years ago

Review comments incorporated in https://github.com/arcprabh/ocp4-upi-powervm/pull/2

@yussufsh - pls check https://github.com/arcprabh/ocp4-upi-powervm/pull/2

yussufsh commented 4 years ago

@arcprabh you need to raise the PR against https://github.com/ocp-power-automation/ocp4-upi-powervm master branch. Not on your forked repo.

And please follow https://github.com/ocp-power-automation/ocp4-upi-powervm/blob/master/CONTRIBUTING.md#pull-request-process

arcprabh commented 4 years ago

@yussufsh

pls review https://github.com/ocp-power-automation/ocp4-upi-powervm/pull/85