search

hpe-storage / csi-driver

A Container Storage Interface (CSI) driver from HPE
https://scod.hpedev.io
Apache License 2.0
58 stars 53 forks source link

disconnecting iscsi volume can break other volumes connections #269

Open krakazyabra opened 3 years ago

krakazyabra commented 3 years ago

If there are more than one volume on one node, there is quite a good chance that when one of them is dissconnected (e.g. pod is removed), it will disconnect iscsi sessions used by other volumes. The result is disconected / read-only / broken fs of other pods running on the same node.

Here is normal state - 4 iscsi connections, used by all volumes: 

root@m6c20:/home/dc145# iscsiadm -m session
tcp: [191] 10.37.120.5:3260,1024 iqn.2000-05.com.3pardata:20210002ac023192 (non-flash)
tcp: [192] 10.37.120.6:3260,1025 iqn.2000-05.com.3pardata:20220002ac023192 (non-flash)
tcp: [193] 10.37.120.7:3260,1026 iqn.2000-05.com.3pardata:21210002ac023192 (non-flash)
tcp: [194] 10.37.120.8:3260,1027 iqn.2000-05.com.3pardata:21220002ac023192 (non-flash)

root@m6c20:/home/dc145# multipath -l
mpathai (360002ac000000000000002a600023192) dm-17 3PARdata,VV
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 2:0:0:4 sde     8:64  active undef running
  |- 3:0:0:4 sdj     8:144 active undef running
  |- 4:0:0:4 sdo     8:224 active undef running
  `- 5:0:0:4 sdt     65:48 active undef running
mpathb (360002ac000000000000002a700023192) dm-14 3PARdata,VV
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 2:0:0:1 sdb     8:16  active undef running
  |- 3:0:0:1 sdg     8:96  active undef running
  |- 4:0:0:1 sdl     8:176 active undef running
  `- 5:0:0:1 sdq     65:0  active undef running
mpathab (360002ac000000000000002a300023192) dm-15 3PARdata,VV
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 2:0:0:2 sdc     8:32  active undef running
  |- 3:0:0:2 sdh     8:112 active undef running
  |- 4:0:0:2 sdm     8:192 active undef running
  `- 5:0:0:2 sdr     65:16 active undef running
mpathu (360002ac000000000000002d300023192) dm-13 3PARdata,VV
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 2:0:0:0 sda     8:0   active undef running
  |- 3:0:0:0 sdf     8:80  active undef running
  |- 4:0:0:0 sdk     8:160 active undef running
  `- 5:0:0:0 sdp     8:240 active undef running
mpaths (360002ac000000000000002a400023192) dm-16 3PARdata,VV
size=150G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  |- 2:0:0:3 sdd     8:48  active undef running
  |- 3:0:0:3 sdi     8:128 active undef running
  |- 4:0:0:3 sdn     8:208 active undef running
  `- 5:0:0:3 sds     65:32 active undef running

And here is the state after one of the pod is killed. All iscsi connections were closed and all volumes are read only, possibly with broken filesystem and data loss: 

root@m6c20:/home/dc145# iscsiadm -m session
iscsiadm: No active sessions.
root@m6c20:/home/dc145# multipath -l
mpathai (360002ac000000000000002a600023192) dm-17 ##,##
size=150G features='0' hwhandler='1 alua' wp=rw
mpathb (360002ac000000000000002a700023192) dm-14 ##,##
size=150G features='0' hwhandler='1 alua' wp=rw
mpathab (360002ac000000000000002a300023192) dm-15 ##,##
size=150G features='0' hwhandler='1 alua' wp=rw
mpaths (360002ac000000000000002a400023192) dm-16 ##,##
size=150G features='0' hwhandler='1 alua' wp=rw

root@m6c20:/home/dc145# mount | grep mpath | grep plugins/hpe
/dev/mapper/mpaths on /var/lib/kubelet/plugins/hpe.com/mounts/pvc-3a300bde-7f12-4141-a9c7-258d94dd4cf8 type ext4 (ro,relatime,stripe=4096)
/dev/mapper/mpathb on /var/lib/kubelet/plugins/hpe.com/mounts/pvc-879270e3-7554-4c78-9b0c-7ce462922078 type ext4 (ro,relatime,stripe=4096)
/dev/mapper/mpathai on /var/lib/kubelet/plugins/hpe.com/mounts/pvc-36f4b7a8-fc9b-455b-8476-441d287859ae type ext4 (ro,relatime,stripe=4096)
/dev/mapper/mpathab on /var/lib/kubelet/plugins/hpe.com/mounts/pvc-4ed3550f-dadc-4f9e-9660-4b9480e6570e type ext4 (ro,relatime,stripe=4096)

root@m6c20:/home/dc145# ls /var/lib/kubelet/plugins/hpe.com/mounts/pvc-3a300bde-7f12-4141-a9c7-258d94dd4cf8
ls: reading directory '/var/lib/kubelet/plugins/hpe.com/mounts/pvc-3a300bde-7f12-4141-a9c7-258d94dd4cf8': Input/output error
datamattsson commented 3 years ago

@sneharai4 can you have a look at this?

sneharai4 commented 3 years ago

@krakazyabra Can you share the multipath -ll output?

krakazyabra commented 3 years ago

Created 2 simple pods with 3par pvc:

Pod

```yaml kind: Pod apiVersion: v1 metadata: name: my-pod-1 namespace: hpe-storage spec: nodeName: m20c20 containers: - name: pod-datelog-1 image: debian command: ["bin/sh"] args: ["-c", "while true; do date >> /data/mydata.txt; sleep 1; done"] volumeMounts: - name: export1 mountPath: /data volumes: - name: export1 persistentVolumeClaim: claimName: my-pvc-1 --- kind: Pod apiVersion: v1 metadata: name: my-pod-2 namespace: hpe-storage spec: nodeName: m20c20 containers: - name: pod-datelog-1 image: debian command: ["bin/sh"] args: ["-c", "while true; do date >> /data/mydata.txt; sleep 1; done"] volumeMounts: - name: export1 mountPath: /data volumes: - name: export1 persistentVolumeClaim: claimName: my-pvc-2 ```

PVC

```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc-1 namespace: hpe-storage spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: 3par --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc-2 namespace: hpe-storage spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: 3par ```

multipath -ll output: 

root@m20c20:/home/ep192# multipath -ll
mpathb (360002ac000000000000002d500023192) dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 2:0:0:1 sdb 8:16  active ready running
  |- 3:0:0:1 sdd 8:48  active ready running
  |- 4:0:0:1 sdf 8:80  active ready running
  `- 5:0:0:1 sdh 8:112 active ready running
mpatha (360002ac000000000000002d400023192) dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 2:0:0:0 sda 8:0   active ready running
  |- 3:0:0:0 sdc 8:32  active ready running
  |- 4:0:0:0 sde 8:64  active ready running
  `- 5:0:0:0 sdg 8:96  active ready running

Deleting one pod:

kubectl delete pod -n hpe-storage my-pod-2

And multipath -ll

root@m20c20:/home/ep192# multipath -ll
mpathb (360002ac000000000000002d500023192) dm-1 ##,##
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw

Screenshot from ssmc

pic

![Screenshot_2021-05-12 m20c20 - Hosts - Primera 3PAR SSMC](https://user-images.githubusercontent.com/26598203/117973588-e06b6680-b32c-11eb-8938-194bfe07c290.png)

krakazyabra commented 3 years ago

Also I collected verbose logs before, in the moment, and after issue:

Before

```shell root@m20c20:/home/ep192# multipath -v3 -ll May 13 10:44:00 | set open fds limit to 20000000/20000000 May 13 10:44:00 | loading //lib/multipath/libchecktur.so checker May 13 10:44:00 | checker tur: message table size = 3 May 13 10:44:00 | loading //lib/multipath/libprioconst.so prioritizer May 13 10:44:00 | foreign library "nvme" loaded successfully May 13 10:44:00 | loop0: device node name blacklisted May 13 10:44:00 | loop1: device node name blacklisted May 13 10:44:00 | loop2: device node name blacklisted May 13 10:44:00 | loop3: device node name blacklisted May 13 10:44:00 | loop4: device node name blacklisted May 13 10:44:00 | loop5: device node name blacklisted May 13 10:44:00 | loop6: device node name blacklisted May 13 10:44:00 | loop7: device node name blacklisted ===== no paths ===== May 13 10:44:00 | libdevmapper version 1.02.167 (2019-11-30) May 13 10:44:00 | DM multipath kernel driver v1.13.0 May 13 10:44:00 | unloading const prioritizer May 13 10:44:00 | unloading tur checker ```

Creating resources

```shell root@m20c20:/home/ep192# multipath -v3 -ll May 13 10:47:47 | set open fds limit to 20000000/20000000 May 13 10:47:47 | loading //lib/multipath/libchecktur.so checker May 13 10:47:47 | checker tur: message table size = 3 May 13 10:47:47 | loading //lib/multipath/libprioconst.so prioritizer May 13 10:47:47 | foreign library "nvme" loaded successfully May 13 10:47:47 | sda: size = 10485760 May 13 10:47:47 | sda: vendor = 3PARdata May 13 10:47:47 | sda: product = VV May 13 10:47:47 | sda: rev = 3313 May 13 10:47:47 | sda: h:b:t:l = 2:0:0:0 May 13 10:47:47 | sda: tgt_node_name = iqn.2000-05.com.3pardata:20210002ac023192 May 13 10:47:47 | sda: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sda: serial = CZ39074677 May 13 10:47:47 | sda: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sda: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sda: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sda: tur state = up May 13 10:47:47 | sdb: size = 10485760 May 13 10:47:47 | sdb: vendor = 3PARdata May 13 10:47:47 | sdb: product = VV May 13 10:47:47 | sdb: rev = 3313 May 13 10:47:47 | sdb: h:b:t:l = 2:0:0:1 May 13 10:47:47 | sdb: tgt_node_name = iqn.2000-05.com.3pardata:20210002ac023192 May 13 10:47:47 | sdb: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdb: serial = CZ39074677 May 13 10:47:47 | sdb: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdb: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdb: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdb: tur state = up May 13 10:47:47 | sdc: size = 10485760 May 13 10:47:47 | sdc: vendor = 3PARdata May 13 10:47:47 | sdc: product = VV May 13 10:47:47 | sdc: rev = 3313 May 13 10:47:47 | sdc: h:b:t:l = 3:0:0:0 May 13 10:47:47 | sdc: tgt_node_name = iqn.2000-05.com.3pardata:20220002ac023192 May 13 10:47:47 | sdc: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdc: serial = CZ39074677 May 13 10:47:47 | sdc: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdc: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdc: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdc: tur state = up May 13 10:47:47 | sdd: size = 10485760 May 13 10:47:47 | sdd: vendor = 3PARdata May 13 10:47:47 | sdd: product = VV May 13 10:47:47 | sdd: rev = 3313 May 13 10:47:47 | sdd: h:b:t:l = 3:0:0:1 May 13 10:47:47 | sdd: tgt_node_name = iqn.2000-05.com.3pardata:20220002ac023192 May 13 10:47:47 | sdd: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdd: serial = CZ39074677 May 13 10:47:47 | sdd: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdd: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdd: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdd: tur state = up May 13 10:47:47 | sde: size = 10485760 May 13 10:47:47 | sde: vendor = 3PARdata May 13 10:47:47 | sde: product = VV May 13 10:47:47 | sde: rev = 3313 May 13 10:47:47 | sde: h:b:t:l = 4:0:0:0 May 13 10:47:47 | sde: tgt_node_name = iqn.2000-05.com.3pardata:21210002ac023192 May 13 10:47:47 | sde: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sde: serial = CZ39074677 May 13 10:47:47 | sde: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sde: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sde: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sde: tur state = up May 13 10:47:47 | sdf: size = 10485760 May 13 10:47:47 | sdf: vendor = 3PARdata May 13 10:47:47 | sdf: product = VV May 13 10:47:47 | sdf: rev = 3313 May 13 10:47:47 | sdf: h:b:t:l = 4:0:0:1 May 13 10:47:47 | sdf: tgt_node_name = iqn.2000-05.com.3pardata:21210002ac023192 May 13 10:47:47 | sdf: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdf: serial = CZ39074677 May 13 10:47:47 | sdf: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdf: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdf: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdf: tur state = up May 13 10:47:47 | sdg: size = 10485760 May 13 10:47:47 | sdg: vendor = 3PARdata May 13 10:47:47 | sdg: product = VV May 13 10:47:47 | sdg: rev = 3313 May 13 10:47:47 | sdg: h:b:t:l = 5:0:0:0 May 13 10:47:47 | sdg: tgt_node_name = iqn.2000-05.com.3pardata:21220002ac023192 May 13 10:47:47 | sdg: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdg: serial = CZ39074677 May 13 10:47:47 | sdg: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdg: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdg: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdg: tur state = up May 13 10:47:47 | sdh: size = 10485760 May 13 10:47:47 | sdh: vendor = 3PARdata May 13 10:47:47 | sdh: product = VV May 13 10:47:47 | sdh: rev = 3313 May 13 10:47:47 | sdh: h:b:t:l = 5:0:0:1 May 13 10:47:47 | sdh: tgt_node_name = iqn.2000-05.com.3pardata:21220002ac023192 May 13 10:47:47 | sdh: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdh: serial = CZ39074677 May 13 10:47:47 | sdh: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdh: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdh: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdh: tur state = up May 13 10:47:47 | loop0: device node name blacklisted May 13 10:47:47 | loop1: device node name blacklisted May 13 10:47:47 | loop2: device node name blacklisted May 13 10:47:47 | loop3: device node name blacklisted May 13 10:47:47 | loop4: device node name blacklisted May 13 10:47:47 | loop5: device node name blacklisted May 13 10:47:47 | loop6: device node name blacklisted May 13 10:47:47 | loop7: device node name blacklisted May 13 10:47:47 | dm-0: device node name blacklisted May 13 10:47:47 | dm-1: device node name blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st 2:0:0:0 sda 8:0 -1 undef undef 3PARdata,VV unknown 2:0:0:1 sdb 8:16 -1 undef undef 3PARdata,VV unknown 3:0:0:0 sdc 8:32 -1 undef undef 3PARdata,VV unknown 3:0:0:1 sdd 8:48 -1 undef undef 3PARdata,VV unknown 4:0:0:0 sde 8:64 -1 undef undef 3PARdata,VV unknown 4:0:0:1 sdf 8:80 -1 undef undef 3PARdata,VV unknown 5:0:0:0 sdg 8:96 -1 undef undef 3PARdata,VV unknown 5:0:0:1 sdh 8:112 -1 undef undef 3PARdata,VV unknown May 13 10:47:47 | libdevmapper version 1.02.167 (2019-11-30) May 13 10:47:47 | DM multipath kernel driver v1.13.0 May 13 10:47:47 | params = 1 queue_if_no_path 1 alua 1 1 service-time 0 4 2 8:16 1 1 8:48 1 1 8:80 1 1 8:112 1 1 May 13 10:47:47 | status = 2 0 1 0 1 1 A 0 4 2 8:16 A 0 0 1 8:48 A 0 0 1 8:80 A 0 0 1 8:112 A 0 0 1 May 13 10:47:47 | sdb: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | loading //lib/multipath/libpriosysfs.so prioritizer May 13 10:47:47 | sdb: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdb: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdb: sysfs prio = 50 May 13 10:47:47 | sdd: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdd: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdd: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdd: sysfs prio = 50 May 13 10:47:47 | sdf: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdf: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdf: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdf: sysfs prio = 50 May 13 10:47:47 | sdh: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdh: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdh: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdh: sysfs prio = 50 mpathd (360002ac000000000000002d600023192) dm-1 3PARdata,VV size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 2:0:0:1 sdb 8:16 active ready running |- 3:0:0:1 sdd 8:48 active ready running |- 4:0:0:1 sdf 8:80 active ready running `- 5:0:0:1 sdh 8:112 active ready running May 13 10:47:47 | params = 1 queue_if_no_path 1 alua 1 1 service-time 0 4 2 8:0 1 1 8:32 1 1 8:64 1 1 8:96 1 1 May 13 10:47:47 | status = 2 0 1 0 1 1 A 0 4 2 8:0 A 0 0 1 8:32 A 0 0 1 8:64 A 0 0 1 8:96 A 0 0 1 May 13 10:47:47 | sda: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sda: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sda: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sda: sysfs prio = 50 May 13 10:47:47 | sdc: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdc: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdc: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdc: sysfs prio = 50 May 13 10:47:47 | sde: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sde: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sde: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sde: sysfs prio = 50 May 13 10:47:47 | sdg: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdg: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdg: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdg: sysfs prio = 50 mpathc (360002ac000000000000002d700023192) dm-0 3PARdata,VV size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 2:0:0:0 sda 8:0 active ready running |- 3:0:0:0 sdc 8:32 active ready running |- 4:0:0:0 sde 8:64 active ready running `- 5:0:0:0 sdg 8:96 active ready running May 13 10:47:47 | unloading sysfs prioritizer May 13 10:47:47 | unloading const prioritizer May 13 10:47:47 | unloading tur checker ```

After deleting

```shell root@m20c20:/home/ep192# multipath -v3 -ll May 13 10:51:41 | set open fds limit to 20000000/20000000 May 13 10:51:41 | loading //lib/multipath/libchecktur.so checker May 13 10:51:41 | checker tur: message table size = 3 May 13 10:51:41 | loading //lib/multipath/libprioconst.so prioritizer May 13 10:51:41 | foreign library "nvme" loaded successfully May 13 10:51:41 | loop0: device node name blacklisted May 13 10:51:41 | loop1: device node name blacklisted May 13 10:51:41 | loop2: device node name blacklisted May 13 10:51:41 | loop3: device node name blacklisted May 13 10:51:41 | loop4: device node name blacklisted May 13 10:51:41 | loop5: device node name blacklisted May 13 10:51:41 | loop6: device node name blacklisted May 13 10:51:41 | loop7: device node name blacklisted May 13 10:51:41 | dm-1: device node name blacklisted ===== no paths ===== May 13 10:51:41 | libdevmapper version 1.02.167 (2019-11-30) May 13 10:51:41 | DM multipath kernel driver v1.13.0 May 13 10:51:41 | params = 1 queue_if_no_path 1 alua 0 0 May 13 10:51:41 | status = 2 0 0 0 0 0 mpathd (360002ac000000000000002d600023192) dm-1 ##,## size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw May 13 10:51:41 | unloading const prioritizer May 13 10:51:41 | unloading tur checker ```

c-snell commented 3 years ago

@krakazyabra - I am unable to recreate this issue on CentOS and Kubernetes 1.19. I am using the HPE CSI Driver 1.4.0. Can you give me some more detail on the Kubernetes version? Host OS? What does your 3PAR multipath.conf look like?

krakazyabra commented 3 years ago

Hello.

multipath.conf on node

```shell root@m20c20:/home/ep192# cat /etc/multipath.conf defaults { user_friendly_names yes } devices { device { features 0 prio alua rr_min_io 100 path_grouping_policy group_by_prio hardware_handler "1 alua" failback immediate path_selector "round-robin 0" product "Server" fast_io_fail_tmo 5 path_checker tur dev_loss_tmo infinity rr_weight uniform vendor "Nimble" no_path_retry 18 } } ```

we're using kubernetes v1.20.5, OS is Ubuntu 20.04.1 LTS Also we use hpe-linux-config

config.json

```yaml kind: ConfigMap apiVersion: v1 metadata: name: hpe-linux-config namespace: hpe-storage data: config.json: |- [ { "category": "iscsi", "severity": "warning", "description": "Manual startup of iSCSI nodes on boot. Can be set in /etc/iscsi/iscsid.conf", "parameter": "startup", "recommendation": "manual" }, { "category": "iscsi", "severity": "warning", "description": "Replacement_timeout of 10 seconds is recommended for faster failover of I/O by multipath on path failures. Can be set in /etc/iscsi/iscsid.conf", "parameter": "replacement_timeout", "recommendation": "10" }, { "category": "iscsi", "severity": "warning", "description": "Minimum login timeout of 15 seconds is recommended with iSCSI. Can be set in /etc/iscsi/iscsid.conf", "parameter": "login_timeout", "recommendation": "15" }, { "category": "iscsi", "severity": "warning", "description": "Minimum timeout of 10 seconds is recommended with noop requests. Can be set in /etc/iscsi/iscsid.conf", "parameter": "noop_out_timeout", "recommendation": "10" }, { "category": "iscsi", "severity": "info", "description": "Minimum cmds_max of 512 is recommended for each session if handling multiple LUN's. Can be set in /etc/iscsi/iscsid.conf", "parameter": "cmds_max", "recommendation": "512" }, { "category": "iscsi", "severity": "warning", "description": "Minimum queue_depth of 256 is recommended for each iSCSI session/path. Can be set in /etc/iscsi/iscsid.conf", "parameter": "queue_depth", "recommendation": "256" }, { "category": "iscsi", "severity": "info", "description": "Minimum number of sessions per iSCSI login is recommended to be 1 by default. If additional sessions are needed this can be set in /etc/iscsi/iscsid.conf. If NCM is running, please change min_session_per_array in /etc/ncm.conf and restart nlt service instead", "parameter": "nr_sessions", "recommendation": "1" }, { "category": "multipath", "severity": "critical", "description": "product attribute recommended to be set to Server in /etc/multipath.conf", "parameter": "product", "recommendation": "\"Server\"" }, { "category": "multipath", "severity": "critical", "description": "alua prioritizer is recommended. Can be set in /etc/multipath.conf", "parameter": "prio", "recommendation": "alua" }, { "category": "multipath", "severity": "critical", "description": "scsi_dh_alua device handler is recommended. Can be set in /etc/multipath.conf", "parameter": "hardware_handler", "recommendation": "\"1 alua\"" }, { "category": "multipath", "severity": "warning", "description": "immediate failback setting is recommended. Can be set in /etc/multipath.conf", "parameter": "failback", "recommendation": "immediate" }, { "category": "multipath", "severity": "critical", "description": "immediately fail i/o on transient path failures to retry on other paths, value=1. Can be set in /etc/multipath.conf", "parameter": "fast_io_fail_tmo", "recommendation": "5" }, { "category": "multipath", "severity": "critical", "description": "queueing is recommended for 150 seconds, with no_path_retry value of 30. Can be set in /etc/multipath.conf", "parameter": "no_path_retry", "recommendation": "18" }, { "category": "multipath", "severity": "warning", "description": "service-time path selector is recommended. Can be set in /etc/multipath.conf", "parameter": "path_selector", "recommendation": "\"round-robin 0\"" }, { "category": "multipath", "severity": "critical", "description": "vendor attribute recommended to be set to Nimble in /etc/multipath.conf", "parameter": "vendor", "recommendation": "\"Nimble\"" }, { "category": "multipath", "severity": "critical", "description": "group paths according to ALUA path priority of active/standby. Recommended to be set to group_by_prio in /etc/multipath.conf", "parameter": "path_grouping_policy", "recommendation": "group_by_prio" }, { "category": "multipath", "severity": "critical", "description": "tur path checker is recommended. Can be set in /etc/multipath.conf", "parameter": "path_checker", "recommendation": "tur" }, { "category": "multipath", "severity": "critical", "description": "infinite value is recommended for timeout in cases of device loss for FC. Can be set in /etc/multipath.conf", "parameter": "dev_loss_tmo", "recommendation": "infinity" }, { "category": "multipath", "severity": "critical", "description": "fine tune 0", "parameter": "features", "recommendation": "0" }, { "category": "multipath", "severity": "critical", "description": "fine tune 1", "parameter": "rr_weight", "recommendation": "uniform" }, { "category": "multipath", "severity": "critical", "description": "fine tune 2", "parameter": "rr_min_io", "recommendation": "100" } ] ```

3par-primera-csp.yaml

```yaml # Configuration to deploy the HPE 3PAR Primera Storage CSP service # # example usage: kubectl create -f ############################################## ############ HPEVolumeInfos CRD ############ ############################################## --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: null name: hpevolumeinfos.storage.hpe.com spec: group: storage.hpe.com names: kind: HPEVolumeInfo plural: hpevolumeinfos shortNames: - hperdi scope: Cluster validation: openAPIV3Schema: properties: hpeVolumes: description: List of HPE volumes configured for 3PAR/Primera arrays. items: properties: uuid: description: The UUID of the volume. type: string record: description: Metadata for the volume. type: map[string]string type: array version: v1 status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- apiVersion: apiextensions.k8s.io/v1beta1 kind: CustomResourceDefinition metadata: creationTimestamp: null name: hpereplicationdeviceinfos.storage.hpe.com spec: group: storage.hpe.com names: kind: HPEReplicationDeviceInfo plural: hpereplicationdeviceinfos scope: Cluster validation: openAPIV3Schema: properties: hpeReplicationDeviceInfos: description: List of HPE Replicated Arrays configured for 3PAR/Primera arrays. items: properties: targets: description: Target Array Details type: array items: properties: targetName: description: Target Name of the array type: string targetCpg: description: Target CPG of the array type: string targetSnapCpg: description: Target Snap CPG of the array type: string targetSecret: description: Secret of the replicated array type: string targetMode: description: Replication Mode type: string targetSecretNamespace: description: Namespace of secret type: string required: - targetName - targetCpg - targetSecret - targetSecretNamespace type: array version: v1 status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- ####################################### ############ CSP Service ############ ####################################### kind: Service apiVersion: v1 metadata: name: primera3par-csp-svc namespace: hpe-storage labels: app: primera3par-csp-svc spec: ports: - port: 8080 protocol: TCP selector: app: primera3par-csp --- ########################################## ############ CSP Deployment ############ ########################################## kind: Deployment apiVersion: apps/v1 metadata: name: primera3par-csp namespace: hpe-storage spec: selector: matchLabels: app: primera3par-csp replicas: 1 template: metadata: labels: app: primera3par-csp spec: nodeSelector: hpe-csi: "" tolerations: - effect: NoSchedule key: node.kubernetes.io/unschedulable operator: 'Exists' serviceAccount: hpe-csp-sa priorityClassName: system-cluster-critical containers: - name: primera3par-csp image: harbor.domain.org/3par/hpe3parprimera-csp@sha256:900b537c5308a39df2a9b950b783d5021c1c08365c1a100c29a8e4f5d599a390 imagePullPolicy: IfNotPresent ports: - containerPort: 8080 volumeMounts: - name: log-dir mountPath: /var/log volumes: - name: log-dir hostPath: path: /var/log ```

hpe-csi-k8s-1.18.yaml

```yaml # Configuration to deploy the HPE CSI driver compatible with # Kubernetes = v1.18 # # example usage: kubectl create -f --- ############################################# ############ HPE Node Info CRD ############ ############################################# apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: creationTimestamp: null name: hpenodeinfos.storage.hpe.com spec: group: storage.hpe.com names: kind: HPENodeInfo plural: hpenodeinfos scope: Cluster versions: - name: v1 # Each version can be enabled/disabled by Served flag. served: true # One and only one version must be marked as the storage version. storage: true schema: openAPIV3Schema: properties: apiVersion: description: "APIVersion defines the versioned schema of this representation of an object." type: string kind: description: "Kind is a string value representing the REST resource this object represents" type: string spec: description: "spec defines the desired characteristics of a HPE nodeinfo requested by a user." properties: chapPassword: description: "The CHAP Password" type: string chapUser: description: "The CHAP User Name" type: string iqns: description: "List of IQNs configured on the node." items: type: string type: array networks: description: "List of networks configured on the node." items: type: string type: array uuid: description: "The UUID of the node." type: string wwpns: description: "List of WWPNs configured on the node." items: type: string type: array required: - uuid - networks type: object required: - spec type: object status: acceptedNames: kind: "" plural: "" conditions: [] storedVersions: [] --- ################# CSI Driver CRD ########### apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: name: csi.hpe.com spec: podInfoOnMount: true volumeLifecycleModes: - Persistent - Ephemeral --- ############################################# ############ Controller driver ############ ############################################# kind: Deployment apiVersion: apps/v1 metadata: name: hpe-csi-controller namespace: hpe-storage spec: replicas: 1 selector: matchLabels: app: hpe-csi-controller template: metadata: labels: app: hpe-csi-controller role: hpe-csi spec: nodeSelector: hpe-csi: "" tolerations: - effect: NoSchedule key: node.kubernetes.io/unschedulable operator: 'Exists' priorityClassName: system-cluster-critical serviceAccount: hpe-csi-controller-sa hostNetwork: true dnsPolicy: ClusterFirstWithHostNet dnsConfig: options: - name: ndots value: "1" containers: - name: csi-provisioner image: harbor.domain.org/3par/csi-provisioner@sha256:6fe39f4682c8677af7761948b196f27c45de566fb5f1612bb8114cd68bfe535e args: - "--csi-address=$(ADDRESS)" - "--v=5" - "--timeout=10s" - "--worker-threads=16" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-attacher image: harbor.domain.org/3par/csi-attacher@sha256:57a991059eb0be8963126d3be841f63e8cb9278692de5659659909fc344157c3 args: - "--v=5" - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-snapshotter image: harbor.domain.org/3par/csi-snapshotter@sha256:458bd2065adde483ff0b6ec0a30f893f0ab3043be8228d9c216989512dc5009b args: - "--v=5" - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-resizer image: harbor.domain.org/3par/csi-resizer@sha256:43195976fb9f94d943f5dd9d58b8afa543be22d09a1165e8a489b7dfe22c657a args: - "--csi-address=$(ADDRESS)" - "--v=5" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: hpe-csi-driver image: harbor.domain.org/3par/csi-driver@sha256:3f31c9479636cd2044dbc7d7072ee7ab31896d2f1aeeb8ad4b6c523444210921 args : - "--endpoint=$(CSI_ENDPOINT)" - "--flavor=kubernetes" - "--pod-monitor" - "--pod-monitor-interval=30" env: - name: CSI_ENDPOINT value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock - name: LOG_LEVEL value: trace imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: log-dir mountPath: /var/log - name: k8s mountPath: /etc/kubernetes - name: hpeconfig mountPath: /etc/hpe-storage - name: root-dir mountPath: /host - name: csi-volume-mutator image: harbor.domain.org/3par/volume-mutator@sha256:464fb8aa8d694ba5a8f63a1f7919da832ac3e22e0a5c15d98f412b38a2effbcf args: - "--v=5" - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi-extensions.sock imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-volume-group-snapshotter image: harbor.domain.org/3par/volume-group-snapshotter@sha256:7e86f846333d58a6c81871fae3bb36fa2888f31de22d1ab8991d6ee31374829a args: - "--v=5" - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi-extensions.sock imagePullPolicy: "Always" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-volume-group-provisioner image: harbor.domain.org/3par/volume-group-provisioner@sha256:221b02007e9c39ca13f5d7eb851e66476cc4966318c296fd54d92f4d1ed6a113 args: - "--v=5" - "--csi-address=$(ADDRESS)" env: - name: ADDRESS value: /var/lib/csi/sockets/pluginproxy/csi-extensions.sock imagePullPolicy: "Always" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ - name: csi-extensions image: harbor.domain.org/3par/csi-extensions@sha256:d7cff6a6a4b016143cb3a250ca19b4608163f9bf65cde7331ed112764f4f2282 args: - "--v=5" - "--endpoint=$(CSI_ENDPOINT)" env: - name: CSI_ENDPOINT value: unix:///var/lib/csi/sockets/pluginproxy/csi-extensions.sock - name: LOG_LEVEL value: trace imagePullPolicy: "IfNotPresent" volumeMounts: - name: socket-dir mountPath: /var/lib/csi/sockets/pluginproxy/ volumes: - name: socket-dir emptyDir: {} - name: log-dir hostPath: path: /var/log - name: k8s hostPath: path: /etc/kubernetes/ - name: hpeconfig hostPath: path: /etc/hpe-storage/ - name: root-dir hostPath: path: / --- kind: ServiceAccount apiVersion: v1 metadata: name: hpe-csi-controller-sa namespace: hpe-storage --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-provisioner-role rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list", "create"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list"] - apiGroups: [""] resources: ["serviceaccounts"] verbs: ["get", "list", "create"] - apiGroups: [""] resources: ["configmaps"] verbs: ["get", "create"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete", "update"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: [""] resources: ["services"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["apps"] resources: ["deployments"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots"] verbs: ["get", "list"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "delete"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "update", "patch", "delete"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-provisioner-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: ClusterRole name: hpe-csi-provisioner-role apiGroup: rbac.authorization.k8s.io --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-attacher-role rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["csinodes"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments"] verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: ["storage.k8s.io"] resources: ["volumeattachments/status"] verbs: ["get", "list", "watch", "update", "create", "delete"] - apiGroups: [""] resources: ["secrets"] verbs: ["get", "watch", "list"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-attacher-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: ClusterRole name: hpe-csi-attacher-role apiGroup: rbac.authorization.k8s.io --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-snapshotter-role rules: - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotclasses"] verbs: ["get", "list", "watch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents/status"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots/status"] verbs: ["update"] - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-snapshotter-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: ClusterRole name: hpe-csi-snapshotter-role apiGroup: rbac.authorization.k8s.io --- # Resizer must be able to work with PVCs, PVs, SCs. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: external-resizer-role rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumeclaims/status"] verbs: ["update", "patch"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-resizer-role subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: ClusterRole name: external-resizer-role apiGroup: rbac.authorization.k8s.io --- # Resizer must be able to work with end point in current namespace # if (and only if) leadership election is enabled kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: hpe-storage name: external-resizer-cfg rules: - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-resizer-role-cfg namespace: hpe-storage subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: Role name: external-resizer-cfg apiGroup: rbac.authorization.k8s.io --- # cluster role to support volumegroup kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-volumegroup-role namespace: hpe-storage rules: - apiGroups: ["storage.hpe.com"] resources: ["volumegroups"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroupcontents"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroupclasses"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroups/status"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroupcontents/status"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list", "create"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims/status"] verbs: ["update", "patch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"] verbs: ["create", "list", "watch", "delete", "get", "update"] - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-volumegroup-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: ClusterRole name: hpe-csi-volumegroup-role apiGroup: rbac.authorization.k8s.io --- # cluster role to support snapshotgroup kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-snapshotgroup-role namespace: hpe-storage rules: - apiGroups: ["storage.hpe.com"] resources: ["snapshotgroups"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["snapshotgroupcontents"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["snapshotgroupclasses"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["snapshotgroups/status"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["snapshotgroupcontents/status"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list", "create"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims/status"] verbs: ["update", "patch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] - apiGroups: ["apiextensions.k8s.io"] resources: ["customresourcedefinitions"] verbs: ["create", "list", "watch", "delete", "get", "update"] - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroups"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroupcontents"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.hpe.com"] resources: ["volumegroupclasses"] verbs: ["get", "list", "watch"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotcontents/status"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots"] verbs: ["create", "get", "list", "watch", "update", "delete"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshots/status"] verbs: ["update"] - apiGroups: ["snapshot.storage.k8s.io"] resources: ["volumesnapshotclasses"] verbs: ["get", "list"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-snapshotgroup-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: ClusterRole name: hpe-csi-snapshotgroup-role apiGroup: rbac.authorization.k8s.io --- # mutator must be able to work with PVCs, PVs, SCs. kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-mutator-role rules: - apiGroups: [""] resources: ["secrets"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "update", "patch"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumeclaims/status"] verbs: ["update", "patch"] - apiGroups: [""] resources: ["events"] verbs: ["list", "watch", "create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-mutator-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa # replace with non-default namespace name namespace: hpe-storage roleRef: kind: ClusterRole name: csi-mutator-role apiGroup: rbac.authorization.k8s.io --- # mutator must be able to work with end point in current namespace # if (and only if) leadership election is enabled kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: namespace: hpe-storage name: csi-mutator-cfg rules: - apiGroups: ["coordination.k8s.io"] resources: ["leases"] verbs: ["get", "watch", "list", "delete", "update", "create"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: csi-mutator-role-cfg namespace: hpe-storage subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage roleRef: kind: Role name: csi-mutator-cfg apiGroup: rbac.authorization.k8s.io --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-driver-role namespace: hpe-storage rules: - apiGroups: ["storage.hpe.com"] resources: ["hpenodeinfos"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] - apiGroups: [""] resources: ["services"] verbs: ["get"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list"] - apiGroups: ["storage.hpe.com"] resources: ["hpevolumeinfos"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["hpereplicationdeviceinfos"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["hpevolumegroupinfos"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: ["storage.hpe.com"] resources: ["hpesnapshotgroupinfos"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] --- ####################################### ############ Node driver ############ ####################################### kind: DaemonSet apiVersion: apps/v1 metadata: name: hpe-csi-node namespace: hpe-storage spec: selector: matchLabels: app: hpe-csi-node template: metadata: labels: app: hpe-csi-node role: hpe-csi spec: nodeSelector: hpe-csi: "" tolerations: - effect: NoSchedule key: node.kubernetes.io/unschedulable operator: 'Exists' priorityClassName: system-node-critical serviceAccount: hpe-csi-node-sa hostNetwork: true dnsPolicy: ClusterFirstWithHostNet dnsConfig: options: - name: ndots value: "1" containers: - name: csi-node-driver-registrar image: harbor.domain.org/3par/csi-node-driver-registrar@sha256:a104f0f0ec5fdd007a4a85ffad95a93cfb73dd7e86296d3cc7846fde505248d3 args: - "--csi-address=$(ADDRESS)" - "--kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)" - "--v=5" env: - name: ADDRESS value: /csi/csi.sock - name: DRIVER_REG_SOCK_PATH value: /var/lib/kubelet/plugins/csi.hpe.com/csi.sock volumeMounts: - name: plugin-dir mountPath: /csi - name: registration-dir mountPath: /registration - name: hpe-csi-driver image: harbor.domain.org/3par/csi-driver@sha256:3f31c9479636cd2044dbc7d7072ee7ab31896d2f1aeeb8ad4b6c523444210921 args : - "--endpoint=$(CSI_ENDPOINT)" - "--node-service" - "--flavor=kubernetes" env: - name: CSI_ENDPOINT value: unix:///csi/csi.sock - name: LOG_LEVEL value: trace - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName - name: CHAP_USER value: "" - name: CHAP_PASSWORD value: "" - name: DISABLE_NODE_CONFORMANCE value: "false" imagePullPolicy: "IfNotPresent" securityContext: privileged: true capabilities: add: ["SYS_ADMIN"] allowPrivilegeEscalation: true volumeMounts: - name: plugin-dir mountPath: /csi - name: pods-mount-dir mountPath: /var/lib/kubelet # needed so that any mounts setup inside this container are # propagated back to the host machine. mountPropagation: "Bidirectional" - name: root-dir mountPath: /host mountPropagation: "Bidirectional" - name: device-dir mountPath: /dev - name: log-dir mountPath: /var/log - name: etc-hpe-storage-dir mountPath: /etc/hpe-storage - name: etc-kubernetes mountPath: /etc/kubernetes - name: sys mountPath: /sys - name: runsystemd mountPath: /run/systemd - name: etcsystemd mountPath: /etc/systemd/system - name: linux-config-file mountPath: /opt/hpe-storage/nimbletune/config.json subPath: config.json volumes: - name: registration-dir hostPath: path: /var/lib/kubelet/plugins_registry type: Directory - name: plugin-dir hostPath: path: /var/lib/kubelet/plugins/csi.hpe.com type: DirectoryOrCreate - name: pods-mount-dir hostPath: path: /var/lib/kubelet - name: root-dir hostPath: path: / - name: device-dir hostPath: path: /dev - name: log-dir hostPath: path: /var/log - name: etc-hpe-storage-dir hostPath: path: /etc/hpe-storage - name: etc-kubernetes hostPath: path: /etc/kubernetes - name: runsystemd hostPath: path: /run/systemd - name: etcsystemd hostPath: path: /etc/systemd/system - name: sys hostPath: path: /sys - name: linux-config-file configMap: name: hpe-linux-config --- apiVersion: v1 kind: ServiceAccount metadata: name: hpe-csi-node-sa namespace: hpe-storage --- kind: ServiceAccount apiVersion: v1 metadata: name: hpe-csp-sa namespace: hpe-storage --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: hpe-csi-driver-binding subjects: - kind: ServiceAccount name: hpe-csi-controller-sa namespace: hpe-storage - kind: ServiceAccount name: hpe-csi-node-sa namespace: hpe-storage - kind: ServiceAccount name: hpe-csp-sa namespace: hpe-storage roleRef: kind: ClusterRole name: hpe-csi-driver-role apiGroup: rbac.authorization.k8s.io ```

replication.yaml

apiVersion: storage.hpe.com/v1 kind: HPEReplicationDeviceInfo metadata: name: replicationdevices spec: target_array_details: - targetCpg: SSD_r6 targetSnapCpg: SSD_r6 targetName: 3par01 targetSecret: 3par01 targetSecretNamespace: hpe-storage ```yaml ```

secrets.yaml

```yaml apiVersion: v1 stringData: serviceName: primera3par-csp-svc servicePort: "8080" backend: <3PAR_IP> username: password: kind: Secret metadata: name: 3par01 namespace: hpe-storage type: Opaque ```

storage_class.yaml

```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: 3par-hosting-1 provisioner: csi.hpe.com parameters: csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/provisioner-secret-name: 3par01 csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: 3par01 csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: 3par01 csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: 3par01 csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage # Uncomment for k8s 1.15 for resize support csi.storage.k8s.io/controller-expand-secret-name: 3par01 csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage # Use pre-existing CPG on 3PAR/Primera cpg: SSD_r6 provisioning_type: tpvv accessProtocol: iscsi allowVolumeExpansion: true reclaimPolicy: Delete ```

c-snell commented 3 years ago

@sneharai4 @raunakkumar This appears to be isolated to how the 3PAR CSP handles volumes with Kubernetes running on Ubuntu nodes. My initial tests were using CentOS nodes and I was unable to recreate the issue. 3PAR/Primera multipathing is part of the multipath driver shipping with RHEL/CentOS.

Upon creating a new cluster using Ubuntu 20.04 with kubeadm 1.20.4 to match customer scenario, I was able to recreate the scenario. I am using the latest HPE CSI Driver 1.4.2 Helm chart.

Here is my output using the same PVCs and Pod YAML files from the customer:

root@kube-ubuntu:~# k get pvc
NAME       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
my-pvc-1   Bound    pvc-05267728-fff2-45ec-a9ac-facc0ba3079b   5Gi        RWO            hpe-standard   34s
my-pvc-2   Bound    pvc-13222920-f614-4749-935d-706751a4e462   5Gi        RWO            hpe-standard   3s
root@kube-ubuntu:~# multipath -ll
root@kube-ubuntu:~# cat /etc/multipath.conf
defaults {
    user_friendly_names yes
}
devices {
    device {
        path_checker         tur
        hardware_handler     "1 alua"
        path_grouping_policy group_by_prio
        no_path_retry        30
        dev_loss_tmo         infinity
        prio                 alua
        failback             immediate
        vendor               "Nimble"
        path_selector        "service-time 0"
        product              "Server"
        fast_io_fail_tmo     5
    }
}
root@kube-ubuntu:~# k get pods
NAME       READY   STATUS    RESTARTS   AGE
my-pod-1   1/1     Running   0          100s
my-pod-2   1/1     Running   0          34s
root@kube-ubuntu:~# multipath -ll
mpathb (360002ac000000000000000dd0007ea94) dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 35:0:0:1 sdg 8:96 active ready running
  |- 34:0:0:1 sdf 8:80 active ready running
  `- 33:0:0:1 sde 8:64 active ready running
mpatha (360002ac000000000000000dc0007ea94) dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 33:0:0:0 sdb 8:16 active ready running
  |- 35:0:0:0 sdd 8:48 active ready running
  `- 34:0:0:0 sdc 8:32 active ready running
root@kube-ubuntu:~# k delete pod my-pod-2
pod "my-pod-2" deleted
root@kube-ubuntu:~# multipath -ll
mpatha (360002ac000000000000000dc0007ea94) dm-1 ##,##
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
root@kube-ubuntu:~# iscsiadm -m session
iscsiadm: No active sessions.
root@kube-ubuntu:~# kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
my-pod-1   1/1     Running   0          21m
root@kube-ubuntu:~# kubectl logs my-pod-1
bin/sh: 1: cannot create /data/mydata.txt: Input/output error
bin/sh: 1: cannot create /data/mydata.txt: Input/output error
sneharai4 commented 3 years ago

@krakazyabra Ubuntu OS is not supported by 3PAR/Primera. Please check the support matrix here, https://scod.hpedev.io/container_storage_provider/hpe_3par_primera/index.html#platform_requirements

c-snell commented 3 years ago

@krakazyabra The issue is our conformance checks haven't applied the proper /etc/multipath.conf settings for Ubuntu and 3PAR. To work around this, you can manually configure the /etc/multipath.conf with the correct settings per HPE 3PAR Ubuntu Operating System Implementation Guide. Reboot the nodes and it should be fixed.

defaults {
    polling_interval 10
    max_fds          8192
}
devices {
    device {
        product              "VV"
        path_grouping_policy group_by_prio
        features             "0"
        prio                 alua
        no_path_retry        18
        hardware_handler     "1 alua"
        failback             immediate
        rr_min_io            100
        vendor               "3PARdata"
        getuid_callout       "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
        path_selector        "round-robin 0"
        path_checker         tur
        checker              tur
    }
    device {
        prio                 alua
        no_path_retry        30
        path_selector        "service-time 0"
        path_checker         tur
        dev_loss_tmo         infinity
        failback             immediate
        fast_io_fail_tmo     5
        path_grouping_policy group_by_prio
        vendor               "Nimble"
        product              "Server"
        hardware_handler     "1 alua"
    }
}
krakazyabra commented 3 years ago

Hello @c-snell Thanks for this workaround. I've added it to my multipath.conf, multipath -t now shows me this additions as well. But problem still exists, when I'm deleting a pod, second exported pvc also disappearing.

multipath -t

```shell defaults { verbosity 2 polling_interval 10 max_polling_interval 40 reassign_maps "no" multipath_dir "//lib/multipath" path_selector "service-time 0" path_grouping_policy "failover" uid_attribute "ID_SERIAL" prio "const" prio_args "" features "0" path_checker "tur" alias_prefix "mpath" failback "manual" rr_min_io 1000 rr_min_io_rq 1 max_fds 8192 rr_weight "uniform" queue_without_daemon "no" flush_on_last_del "no" user_friendly_names "no" fast_io_fail_tmo 5 bindings_file "/etc/multipath/bindings" wwids_file "/etc/multipath/wwids" prkeys_file "/etc/multipath/prkeys" log_checker_err always all_tg_pt "no" retain_attached_hw_handler "yes" detect_prio "yes" detect_checker "yes" force_sync "no" strict_timing "no" deferred_remove "no" config_dir "/etc/multipath/conf.d" delay_watch_checks "no" delay_wait_checks "no" san_path_err_threshold "no" san_path_err_forget_rate "no" san_path_err_recovery_time "no" marginal_path_err_sample_time "no" marginal_path_err_rate_threshold "no" marginal_path_err_recheck_gap_time "no" marginal_path_double_failed_time "no" find_multipaths "on" uxsock_timeout 4000 retrigger_tries 0 retrigger_delay 10 missing_uev_wait_timeout 30 skip_kpartx "no" disable_changed_wwids ignored remove_retries 0 ghost_delay "no" find_multipaths_timeout -10 enable_foreign "" marginal_pathgroups "no" } blacklist { devnode "^drbd[0-9]+" devnode "^(ram|zram|raw|loop|fd|md|dm-|sr|scd|st|dcssblk)[0-9]" devnode "^(td|hd|vd)[a-z]" devnode "^cciss!c[0-9]d[0-9]*" device { vendor "SGI" product "Universal Xport" } device { vendor "^DGC" product "LUNZ" } device { vendor "EMC" product "LUNZ" } device { vendor "DELL" product "Universal Xport" } device { vendor "IBM" product "Universal Xport" } device { vendor "IBM" product "S/390" } device { vendor "LENOVO" product "Universal Xport" } device { vendor "(NETAPP|LSI|ENGENIO)" product "Universal Xport" } device { vendor "STK" product "Universal Xport" } device { vendor "SUN" product "Universal Xport" } device { vendor "(Intel|INTEL)" product "VTrak V-LUN" } device { vendor "Promise" product "VTrak V-LUN" } device { vendor "Promise" product "Vess V-LUN" } } blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } devices { device { vendor "NVME" product ".*" uid_attribute "ID_WWN" path_checker "none" retain_attached_hw_handler "no" } device { vendor "APPLE" product "Xserve RAID" path_grouping_policy "multibus" } device { vendor "DEC" product "HSG80" path_grouping_policy "group_by_prio" path_checker "hp_sw" hardware_handler "1 hp_sw" prio "hp_sw" no_path_retry "queue" } device { vendor "HP" product "A6189A" path_grouping_policy "multibus" no_path_retry 12 } device { vendor "(COMPAQ|HP)" product "(MSA|HSV)1[01]0" path_grouping_policy "group_by_prio" path_checker "hp_sw" hardware_handler "1 hp_sw" prio "hp_sw" no_path_retry 12 } device { vendor "(COMPAQ|HP)" product "MSA VOLUME" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "(COMPAQ|HP)" product "(HSV1[01]1|HSV2[01]0|HSV3[046]0|HSV4[05]0)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "HP" product "(MSA2[02]12fc|MSA2012i)" path_grouping_policy "multibus" no_path_retry 18 } device { vendor "HP" product "(MSA2012sa|MSA23(12|24)(fc|i|sa)|MSA2000s VOLUME)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "HP" product "MSA [12]0[45]0 SA[NS]" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "HP" product "HSVX700" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "HP" product "LOGICAL VOLUME" path_grouping_policy "multibus" no_path_retry 12 } device { vendor "HP" product "(P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "LEFTHAND" product "(P4000|iSCSIDisk|FCDISK)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "SGI" product "TP9100" path_grouping_policy "multibus" } device { vendor "SGI" product "TP9[3457]00" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SGI" product "IS" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SGI" product "^DD[46]A-" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "DDN" product "SAN DataDirector" path_grouping_policy "multibus" } device { vendor "DDN" product "^EF3010" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "DDN" product "^(EF3015|S2A|SFA)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "NEXENTA" product "COMSTAR" path_grouping_policy "group_by_serial" no_path_retry 30 } device { vendor "TEGILE" product "(ZEBI-(FC|ISCSI)|INTELLIFLASH)" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 10 } device { vendor "EMC" product "SYMMETRIX" path_grouping_policy "multibus" no_path_retry 6 } device { vendor "^DGC" product "^(RAID|DISK|VRAID)" product_blacklist "LUNZ" path_grouping_policy "group_by_prio" path_checker "emc_clariion" hardware_handler "1 emc" prio "emc" failback "immediate" no_path_retry 60 } device { vendor "EMC" product "Invista" product_blacklist "LUNZ" path_grouping_policy "multibus" no_path_retry 5 } device { vendor "XtremIO" product "XtremApp" path_grouping_policy "multibus" } device { vendor "COMPELNT" product "Compellent Vol" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "DELL" product "^MD3" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "NVME" product "^EMC PowerMax_" path_grouping_policy "multibus" } device { vendor "FSC" product "CentricStor" path_grouping_policy "group_by_serial" } device { vendor "FUJITSU" product "ETERNUS_DX(H|L|M|400|8000)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 10 } device { vendor "(EUROLOGC|EuroLogc)" product "FC2502" path_grouping_policy "multibus" } device { vendor "FUJITSU" product "E[234]000" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 10 } device { vendor "FUJITSU" product "E[68]000" path_grouping_policy "multibus" no_path_retry 10 } device { vendor "(HITACHI|HP)" product "^OPEN-" path_grouping_policy "multibus" } device { vendor "HITACHI" product "^DF" path_grouping_policy "group_by_prio" prio "hds" failback "immediate" no_path_retry "queue" } device { vendor "HITACHI" product "^DF600F" path_grouping_policy "multibus" } device { vendor "IBM" product "ProFibre 4000R" path_grouping_policy "multibus" } device { vendor "IBM" product "^1722-600" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1724" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1726" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1742" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1746" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1813" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1814" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1815" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1818" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^3526" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^(3542|3552)" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^2105" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "IBM" product "^1750500" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "IBM" product "^2107900" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "IBM" product "^2145" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "IBM" product "S/390 DASD ECKD" product_blacklist "S/390" path_grouping_policy "multibus" uid_attribute "ID_UID" path_checker "directio" no_path_retry "queue" } device { vendor "IBM" product "S/390 DASD FBA" product_blacklist "S/390" path_grouping_policy "multibus" uid_attribute "ID_UID" path_checker "directio" no_path_retry "queue" } device { vendor "IBM" product "^IPR" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "IBM" product "1820N00" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "(XIV|IBM)" product "(NEXTRA|2810XIV)" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "(TMS|IBM)" product "(RamSan|FlashSystem)" path_grouping_policy "multibus" } device { vendor "IBM" product "^(DCS9900|2851)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "AIX" product "VDASD" path_grouping_policy "multibus" no_path_retry 60 } device { vendor "IBM" product "3303[ ]+NVDISK" no_path_retry 60 } device { vendor "AIX" product "NVDISK" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 60 } device { vendor "LENOVO" product "DE_Series" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "NETAPP" product "LUN" path_grouping_policy "group_by_prio" features "2 pg_init_retries 50" prio "ontap" failback "immediate" no_path_retry "queue" flush_on_last_del "yes" dev_loss_tmo "infinity" user_friendly_names "no" } device { vendor "(NETAPP|LSI|ENGENIO)" product "INF-01-00" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SolidFir" product "SSD SAN" path_grouping_policy "multibus" no_path_retry 24 } device { vendor "NVME" product "^NetApp ONTAP Controller" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "NEC" product "DISK ARRAY" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" } device { vendor "^Pillar" product "^Axiom" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" } device { vendor "^Oracle" product "^Oracle FS" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" } device { vendor "STK" product "BladeCtlr" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "STK" product "OPENstorage" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "STK" product "FLEXLINE 380" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "StorEdge 3" path_grouping_policy "multibus" } device { vendor "SUN" product "STK6580_6780" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "CSM[12]00_R" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "LCSM100_[IEFS]" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "SUN_6180" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "ArrayStorage" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "(Sun Storage|ZFS Storage|COMSTAR)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "PIVOT3" product "RAIGE VOLUME" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "(NexGen|Pivot3)" product "(TierStore|vSTAC)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "(Intel|INTEL)" product "Multi-Flex" product_blacklist "VTrak V-LUN" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "(LIO-ORG|SUSE)" product "RBD" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "DataCore" product "SANmelody" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "DataCore" product "Virtual Disk" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "PURE" product "FlashArray" path_grouping_policy "multibus" } device { vendor "HUAWEI" product "XSG1" path_grouping_policy "group_by_prio" prio "alua" } device { vendor "KOVE" product "XPD" path_grouping_policy "multibus" } device { vendor "NFINIDAT" product "InfiniBox" path_grouping_policy "group_by_prio" path_selector "round-robin 0" prio "alua" failback 30 rr_weight "priorities" no_path_retry "fail" rr_min_io 1 rr_min_io_rq 1 flush_on_last_del "yes" fast_io_fail_tmo 15 dev_loss_tmo 15 } device { vendor "KMNRIO" product "K2" path_grouping_policy "multibus" } device { vendor "NEXSAN" product "NXS-B0" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 15 } device { vendor "NEXSAN" product "SATAB" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 15 } device { vendor "Nexsan" product "(NestOS|NST5000)" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "VIOLIN" product "SAN ARRAY$" path_grouping_policy "group_by_serial" no_path_retry 30 } device { vendor "VIOLIN" product "SAN ARRAY ALUA" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "VIOLIN" product "CONCERTO ARRAY" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "(XIOTECH|XIOtech)" product "ISE" path_grouping_policy "multibus" no_path_retry 12 } device { vendor "(XIOTECH|XIOtech)" product "IGLU DISK" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "(XIOTECH|XIOtech)" product "Magnitude" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "Promise" product "VTrak" product_blacklist "VTrak V-LUN" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "Promise" product "Vess" product_blacklist "Vess V-LUN" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "^IFT" product ".*" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "DotHill" product "SANnet" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "DotHill" product "R/Evo" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "DotHill" product "^DH" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "AStor" product "NeoSapphire" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "INSPUR" product "MCS" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" } device { vendor "3PARdata" product "VV" path_grouping_policy "group_by_prio" getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" path_selector "round-robin 0" path_checker "tur" features "0" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 18 rr_min_io 100 fast_io_fail_tmo 10 dev_loss_tmo "infinity" } device { vendor "Nimble" product "Server" path_grouping_policy "group_by_prio" path_selector "service-time 0" path_checker "tur" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 fast_io_fail_tmo 5 dev_loss_tmo "infinity" } } overrides { } ```

c-snell commented 3 years ago

@krakazyabra Hmm, I am not experiencing the volume disconnects after changing the /etc/multipath.conf. I have a similar output to your multipath -t.

Can you share a copy of the updated multipath.conf? Did you reboot your nodes after updating the /etc/multipath.conf?

Also have you opened a support case for this?

HPE Alletra 9000, Primera and 3PAR Hardware Contract Type Phone Number
Warranty and Foundation Care 800-633-3600
Proactive Care (PC) 866-211-5211
Datacenter Care (DC) 888-751-2149
krakazyabra commented 3 years ago

Hello, @c-snell multipath.conf

root@m20c20:/home/ep192# cat /etc/multipath.conf 
defaults {
    polling_interval 10
    max_fds          8192
}
devices {
    device {
        product              "VV"
        path_grouping_policy group_by_prio
        features             "0"
        prio                 alua
        no_path_retry        18
        hardware_handler     "1 alua"
        failback             immediate
        rr_min_io            100
        vendor               "3PARdata"
        getuid_callout       "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
        path_selector        "round-robin 0"
        path_checker         tur
        checker              tur
    }
    device {
        prio                 alua
        no_path_retry        30
        path_selector        "service-time 0"
        path_checker         tur
        dev_loss_tmo         infinity
        failback             immediate
        fast_io_fail_tmo     5
        path_grouping_policy group_by_prio
        vendor               "Nimble"
        product              "Server"
        hardware_handler     "1 alua"
    }
}

No, I didn't make restart, only reload multipathd daemon. We're using pxe boot for all our servers, so this config should be included into our global pxe image. And this is not so fast)

krakazyabra commented 3 years ago

Can confirm: after updating base PXE image and including multipath config there, problem still presents, even after reboot.

krakazyabra commented 3 years ago

Hi! I've updated to 2.0.0 release. I was using same pods and pvc. Here are multipath -ll -v 2 reports.

Create 2 pods with pvc: 

root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f000023192 dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 2:0:0:2 sdc 8:32  active ready running
  |- 3:0:0:2 sdg 8:96  active ready running
  `- 4:0:0:2 sdk 8:160 active ready running
360002ac000000000000002f100023192 dm-3 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 5:0:0:3 sdp 8:240 active ready running
  |- 2:0:0:3 sdd 8:48  active ready running
  |- 3:0:0:3 sdh 8:112 active ready running
  `- 4:0:0:3 sdl 8:176 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 5:0:0:1 sdn 8:208 active ready running
  |- 2:0:0:1 sdb 8:16  active ready running
  |- 3:0:0:1 sdf 8:80  active ready running
  `- 4:0:0:1 sdj 8:144 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 5:0:0:0 sdm 8:192 active ready running
  |- 2:0:0:0 sda 8:0   active ready running
  |- 3:0:0:0 sde 8:64  active ready running
  `- 4:0:0:0 sdi 8:128 active ready running

Delete pod my-pod-1:

root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f100023192 dm-3 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:3 sdl 8:176 active ready running
  `- 5:0:0:3 sdp 8:240 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:1 sdj 8:144 active ready running
  `- 5:0:0:1 sdn 8:208 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:0 sdi 8:128 active ready running
  `- 5:0:0:0 sdm 8:192 active ready running

This is much better! At least, not all connections were unexport.

Create pod my-pod-1 back: 

root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f000023192 dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:2 sdk 8:160 active ready running
  |- 5:0:0:2 sdo 8:224 active ready running
  |- 2:0:0:2 sdc 8:32  active ready running
  `- 3:0:0:2 sdg 8:96  active ready running
360002ac000000000000002f100023192 dm-3 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:3 sdl 8:176 active ready running
  |- 5:0:0:3 sdp 8:240 active ready running
  |- 2:0:0:3 sdd 8:48  active ready running
  `- 3:0:0:3 sdh 8:112 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:1 sdj 8:144 active ready running
  |- 5:0:0:1 sdn 8:208 active ready running
  |- 2:0:0:1 sdb 8:16  active ready running
  `- 3:0:0:1 sdf 8:80  active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 4:0:0:0 sdi 8:128 active ready running
  |- 5:0:0:0 sdm 8:192 active ready running
  |- 2:0:0:0 sda 8:0   active ready running
  `- 3:0:0:0 sde 8:64  active ready running

And now delete my-pod-2

root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f000023192 dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  `- 5:0:0:2 sdo 8:224 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  `- 5:0:0:1 sdn 8:208 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  `- 5:0:0:0 sdm 8:192 active ready running

I cannot say, that it's working perfect, because several times I saw, that all connections were unexported. But it's much better than in beginning.

c-snell commented 3 years ago

@krakazyabra - Glad to hear it is working better. There have been a number of improvements done on the driver with regards to 3PAR/Primera. As of now the engineering team has not validated the HPE CSI Driver on Ubuntu nodes using HPE Alletra 9000, Primera, or 3PAR storage due to limited support of Ubuntu by the storage arrays. Here is the current support matrix: Platform Requirements. All validations have been done with RHEL, CentOS, or SLES. Full support for Ubuntu OS with HPE Primera and Alletra is slated for later this year and will then be validated on the HPE CSI Driver in an upcoming release.

krakazyabra commented 3 years ago

I've made one more test, more complicated. We have another 3par (the same model). So, we connected it and deployed new storageClass, using the same csi-driver (just few new yamls)

replication.yaml

apiVersion: storage.hpe.com/v1
kind: HPEReplicationDeviceInfo
metadata:
  name: replicationdevices-3par02
spec:
  target_array_details:
  - targetCpg: SSD_r6
    targetSnapCpg: SSD_r6
    targetName: 3par02
    targetSecret: 3par02
    targetSecretNamespace: hpe-storage

secrets.yaml

apiVersion: v1
stringData:
  serviceName: primera3par-csp-svc
  servicePort: "8080"
  backend: 10.36.115.11
  username: username
  password: password
kind: Secret
metadata:
  name: 3par02
  namespace: hpe-storage
type: Opaque

storage_class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: 3par02
provisioner: csi.hpe.com
parameters:
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/provisioner-secret-name: 3par02
  csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage
  csi.storage.k8s.io/controller-publish-secret-name: 3par02
  csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage
  csi.storage.k8s.io/node-stage-secret-name: 3par02
  csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage
  csi.storage.k8s.io/node-publish-secret-name: 3par02
  csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage
  # Uncomment for k8s 1.15 for resize support
  csi.storage.k8s.io/controller-expand-secret-name: 3par02
  csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage
  # Use pre-existing CPG on 3PAR/Primera
  cpg: SSD_r6
  provisioning_type: tpvv
  accessProtocol: iscsi
  # remoteCopyGroup: k8sRemoteCopyGroup
  # replicationDevices: replicationdevices
allowVolumeExpansion: true
reclaimPolicy: Delete

And created 2 pods. my-pod-1 has pvc from storageClass 3par, but second pod now has pvc from 3par02 storageClass. Then I deleted one pod, as usual. And what do you think? Csi-drivered worked as expected! It disconnect only volume from deleted node.

create 2 pods: 

root@m20c20:/home/ep192/open-iscsi/iscsiuio# multipath -ll -v 2
360002ac0000000000000009900019d4a dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 2:0:0:0 sda 8:0   active ready running
  |- 3:0:0:0 sdb 8:16  active ready running
  |- 4:0:0:0 sdc 8:32  active ready running
  `- 6:0:0:0 sdd 8:48  active ready running
360002ac000000000000002f200023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 5:0:0:0 sde 8:64  active ready running
  |- 7:0:0:0 sdf 8:80  active ready running
  |- 8:0:0:0 sdg 8:96  active ready running
  `- 9:0:0:0 sdh 8:112 active ready running

delete my-pod-1:

root@m20c20:/home/ep192/open-iscsi/iscsiuio# multipath -ll -v 2
360002ac0000000000000009900019d4a dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 2:0:0:0 sda 8:0  active ready running
  |- 3:0:0:0 sdb 8:16 active ready running
  |- 4:0:0:0 sdc 8:32 active ready running
  `- 6:0:0:0 sdd 8:48 active ready running

Run this test several times. And result always is the same.

IDK, but may be this info will help you somehow to speed up new release for Ubuntu ;-)

krakazyabra commented 3 years ago

it seems we're using Groups Scoped Targets, but driver thinks that we're using Volume Scoped Targets.

image source: https://newman.cloud/en/2019/05/28/sync-replication-and-peer-persistence-setup-hpe-nimble/

@kvaps tried to improve driver's logic and we found, that with this changes driver doesn't make logout from target after removing only one volume.

I performed a lot of testings and each test was good. Only one volume was deleting at once. All another were working well without any I/O errors.

some tests:

Create 2 pods with pvc (from messages below)

root@m20c2:/home/ep192# multipath -ll -v 2
mpathh (360002ac000000000000000c700019d4a) dm-11 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 3:0:0:0 sdf     8:80  active ready running
  |- 4:0:0:0 sdg     8:96  active ready running
  |- 5:0:0:0 sdh     8:112 active ready running
  `- 2:0:0:0 sde     8:64  active ready running
mpathg (360002ac000000000000000c600019d4a) dm-10 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 3:0:0:1 sdb     8:16  active ready running
  |- 4:0:0:1 sdc     8:32  active ready running
  |- 5:0:0:1 sdd     8:48  active ready running
  `- 2:0:0:1 sda     8:0   active ready running

Delete one pod

root@m20c2:/home/ep192# multipath -ll -v 2
mpathh (360002ac000000000000000c700019d4a) dm-11 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
  |- 3:0:0:0 sdf     8:80  active ready running
  |- 4:0:0:0 sdg     8:96  active ready running
  |- 5:0:0:0 sdh     8:112 active ready running
  `- 2:0:0:0 sde     8:64  active ready running

and 

root@m20c2:/home/ep192# iscsiadm -m session -P3
iSCSI Transport Class version 2.0-870
version 2.0-874
Target: iqn.2000-05.com.3pardata:20220002ac019d4a (non-flash)
        Current Portal: 10.37.120.10:3260,1024
        Persistent Portal: 10.37.120.10:3260,1024
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
                Iface IPaddress: 10.37.131.151
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 10
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 10
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 16384
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 3  State: running
                scsi3 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdf          State: running
                scsi3 Channel 00 Id 0 Lun: 254
Target: iqn.2000-05.com.3pardata:21210002ac019d4a (non-flash)
        Current Portal: 10.37.120.11:3260,1025
        Persistent Portal: 10.37.120.11:3260,1025
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
                Iface IPaddress: 10.37.131.151
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 11
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 10
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 16384
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 4  State: running
                scsi4 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdg          State: running
                scsi4 Channel 00 Id 0 Lun: 254
Target: iqn.2000-05.com.3pardata:21220002ac019d4a (non-flash)
        Current Portal: 10.37.120.12:3260,1027
        Persistent Portal: 10.37.120.12:3260,1027
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
                Iface IPaddress: 10.37.131.151
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 12
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 10
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 16384
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 5  State: running
                scsi5 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sdh          State: running
                scsi5 Channel 00 Id 0 Lun: 254
Target: iqn.2000-05.com.3pardata:20210002ac019d4a (non-flash)
        Current Portal: 10.37.120.9:3260,1026
        Persistent Portal: 10.37.120.9:3260,1026
                **********
                Interface:
                **********
                Iface Name: default
                Iface Transport: tcp
                Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
                Iface IPaddress: 10.37.131.151
                Iface HWaddress: <empty>
                Iface Netdev: <empty>
                SID: 9
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
                *********
                Timeouts:
                *********
                Recovery Timeout: 10
                Target Reset Timeout: 30
                LUN Reset Timeout: 30
                Abort Timeout: 15
                *****
                CHAP:
                *****
                username: <empty>
                password: ********
                username_in: <empty>
                password_in: ********
                ************************
                Negotiated iSCSI params:
                ************************
                HeaderDigest: None
                DataDigest: None
                MaxRecvDataSegmentLength: 262144
                MaxXmitDataSegmentLength: 65536
                FirstBurstLength: 16384
                MaxBurstLength: 262144
                ImmediateData: Yes
                InitialR2T: Yes
                MaxOutstandingR2T: 1
                ************************
                Attached SCSI devices:
                ************************
                Host Number: 2  State: running
                scsi2 Channel 00 Id 0 Lun: 0
                        Attached scsi disk sde          State: running
                scsi2 Channel 00 Id 0 Lun: 254

As you see, luns were deleted, but sessions to 3par are still alive. We think, that driver should perform logout operation only if there are no more active luns (exports).

paka86 commented 3 years ago

@c-snell I am having exactly same issue on OpenShift 4.6 with v2.0.0 plugin version for 3Par so I don't think the problem is limited to Ubuntu only. I run multiple tests and often all iscsi sessions gets terminated once the PV is getting unmounted.

ondrejlinha commented 2 years ago

@c-snell We're facing exactly the same issue as @paka86 & @krakazyabra

OpenShift 4.6.44 / HPE CSI Driver 1.4.0

krakazyabra commented 2 years ago

@paka86 @ondrejlinha try this version of driver (hpe-csi-driver): docker pull ghcr.io/kvaps/hpestorage-csi-driver@sha256:c416404f6b3069edf1a409a1f637a43cf1a267c14866cc9173a784134cd37cb2

It has this changes. For us it completelly resolved all problems

datamattsson commented 2 years ago

@krakazyabra this has been fixed in this commit.