Execution of iscsi Mulltipathing

Here are the steps to enable iscsi multipathing :

We will need #173 to expose the tgt targets in every tgt servers. For example : the file /etc/tgt/conf.d/kumo-dan-installscript-img49.conf with the below configuration will have to be replicated.

<target try>
driver iscsi
bs-type rbd
backing-store bmi-introspect/kumo-dan-installscript-img49
bsopts ""conf=/etc/ceph/ceph.conf;id=bmi-introspect""
write-cache off
initiator-address ALL
</target>

*2. Make changes in the /var/lib/tftpboot/.ipxe File (Add the 2nd target in the ipxe file)** For Example :

sleep 23
set keep-san 1
ifconf --configurator=dhcp net0
sanboot iscsi:10.20.30.1:tcp:3260:1:kumo-dan-installscript-img37 \
            iscsi:10.20.30.2:tcp:3260:1:kumo-dan-installscript-img37
boot

wherein 10.20.30.1 and 10..20.30.2 are the IP addresses of the respective tgt servers.

*3. Boot the node with the above change(Reprovision the node)* On booting the node will have the entry of both the tgt servers in the /sys/firmware/ibft

[root@localhost ~]# cat /sys/firmware/ibft/target0/ip-addr 
10.20.30.1
[root@localhost ~]# cat /sys/firmware/ibft/target1/ip-addr 
10.20.30.2
[root@localhost ~]# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0 48.8G  0 disk 
├─sda1   8:1    0  476M  0 part 
└─sda2   8:2    0 48.4G  0 part 
sdb      8:16   0 48.8G  0 disk 
├─sdb1   8:17   0  476M  0 part /boot
└─sdb2   8:18   0 48.4G  0 part /

Now a shell script which performs the following can be executed:
yum update
yum install device-mapper-multipath
mpathconf --enable --with_multipathd y
mpathconf --enable

Make the following change in /etc/multipath.conf : (uncommenting the defaults settings)

defaults {
    polling_interval        10
    path_selector           "round-robin 0"
    path_grouping_policy    multibus
    uid_attribute           ID_SERIAL
#       prio                    alua
    path_checker            readsector0
    rr_min_io               100
    max_fds                 8192
    rr_weight               priorities
    failback                immediate
    no_path_retry           5
    user_friendly_names     yes
    find_multipaths         yes
}

systemctl start multipathd; systemctl enable multipathd
Reboot

on logging into the node again: another shell script will have to be executed that does the following :

1.multipath -a /dev/sdb (which is where my boot drive is)
2.dracut --force -H --add multipath
3.shutdown
4.reboot

Multipathing gets established and successful hand off takes place in the event of any of the tgt servers going down. This is how it looks like once we log inn to the node:

[root@localhost ~]# multipath -l
mpatha (360000000000000000e00000000010001) dm-0 IET     ,VIRTUAL-DISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  |- 7:0:0:1 sda 8:0  active undef unknown
  `- 8:0:0:1 sdb 8:16 active undef unknown

[root@localhost ~]# multipath -ll
mpatha (360000000000000000e00000000010001) dm-0 IET     ,VIRTUAL-DISK
size=10G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 7:0:0:1 sda 8:0  active ready running
  `- 8:0:0:1 sdb 8:16 active ready running

[root@localhost ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda           8:0    0   10G  0 disk
+-mpatha    253:0    0   10G  0 mpath
  +-mpatha1 253:1    0  476M  0 part  /boot
  +-mpatha2 253:2    0    5G  0 part  /
sdb           8:16   0   10G  0 disk
+-mpatha    253:0    0   10G  0 mpath
  +-mpatha1 253:1    0  476M  0 part  /boot
  +-mpatha2 253:2    0    5G  0 part  /

At this point

A deep snapshot of the node from BMI can be taken to provision other nodes with multipathing enabled over it.
"N" TGT servers can be multipathed by simply discovering and logging into them using adm commands, those targets automatically get multipathed.

Problems with this approach :

Automating the shell scripts as two reboots are needed
Both the TGT servers are needed to be up for booting the node, otherwise the node does not boot up and gets stuck in dracut. https://docs.oracle.com/cd/E50245_01/E50246/html/ch06s101.html

As a solution to the above problem, multipathing with three tgt servers instead of two had been suggested by @SahilTikale and @apoorvemohan , that would look like this:

  sleep 23
  set keep-san 1
  ifconf --configurator=dhcp net0
  sanboot iscsi:**10.20.30.1**:tcp:3260:1:kumo-dan-installscript-img37 \
                iscsi:**10.20.30.2**:tcp:3260:1:kumo-dan-installscript-img37  \
                iscsi:**10.20.30.3**:tcp:3260:1:kumo-dan-installscript-img37
  boot

However, There seems to be some line limit on the numbers of arguments that can be passed in the ipxe file. #176

CCI-MOC / m2

Execution of iscsi Mulltipathing #175