Open krakazyabra opened 3 years ago
@sneharai4 can you have a look at this?
@krakazyabra Can you share the multipath -ll output?
Created 2 simple pods with 3par pvc:
```yaml kind: Pod apiVersion: v1 metadata: name: my-pod-1 namespace: hpe-storage spec: nodeName: m20c20 containers: - name: pod-datelog-1 image: debian command: ["bin/sh"] args: ["-c", "while true; do date >> /data/mydata.txt; sleep 1; done"] volumeMounts: - name: export1 mountPath: /data volumes: - name: export1 persistentVolumeClaim: claimName: my-pvc-1 --- kind: Pod apiVersion: v1 metadata: name: my-pod-2 namespace: hpe-storage spec: nodeName: m20c20 containers: - name: pod-datelog-1 image: debian command: ["bin/sh"] args: ["-c", "while true; do date >> /data/mydata.txt; sleep 1; done"] volumeMounts: - name: export1 mountPath: /data volumes: - name: export1 persistentVolumeClaim: claimName: my-pvc-2 ```
```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc-1 namespace: hpe-storage spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: 3par --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: my-pvc-2 namespace: hpe-storage spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi storageClassName: 3par ```
multipath -ll
output:
root@m20c20:/home/ep192# multipath -ll
mpathb (360002ac000000000000002d500023192) dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 2:0:0:1 sdb 8:16 active ready running
|- 3:0:0:1 sdd 8:48 active ready running
|- 4:0:0:1 sdf 8:80 active ready running
`- 5:0:0:1 sdh 8:112 active ready running
mpatha (360002ac000000000000002d400023192) dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 2:0:0:0 sda 8:0 active ready running
|- 3:0:0:0 sdc 8:32 active ready running
|- 4:0:0:0 sde 8:64 active ready running
`- 5:0:0:0 sdg 8:96 active ready running
Deleting one pod:
kubectl delete pod -n hpe-storage my-pod-2
And multipath -ll
root@m20c20:/home/ep192# multipath -ll
mpathb (360002ac000000000000002d500023192) dm-1 ##,##
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
Screenshot from ssmc
![Screenshot_2021-05-12 m20c20 - Hosts - Primera 3PAR SSMC](https://user-images.githubusercontent.com/26598203/117973588-e06b6680-b32c-11eb-8938-194bfe07c290.png)
Also I collected verbose logs before, in the moment, and after issue:
```shell root@m20c20:/home/ep192# multipath -v3 -ll May 13 10:44:00 | set open fds limit to 20000000/20000000 May 13 10:44:00 | loading //lib/multipath/libchecktur.so checker May 13 10:44:00 | checker tur: message table size = 3 May 13 10:44:00 | loading //lib/multipath/libprioconst.so prioritizer May 13 10:44:00 | foreign library "nvme" loaded successfully May 13 10:44:00 | loop0: device node name blacklisted May 13 10:44:00 | loop1: device node name blacklisted May 13 10:44:00 | loop2: device node name blacklisted May 13 10:44:00 | loop3: device node name blacklisted May 13 10:44:00 | loop4: device node name blacklisted May 13 10:44:00 | loop5: device node name blacklisted May 13 10:44:00 | loop6: device node name blacklisted May 13 10:44:00 | loop7: device node name blacklisted ===== no paths ===== May 13 10:44:00 | libdevmapper version 1.02.167 (2019-11-30) May 13 10:44:00 | DM multipath kernel driver v1.13.0 May 13 10:44:00 | unloading const prioritizer May 13 10:44:00 | unloading tur checker ```
```shell root@m20c20:/home/ep192# multipath -v3 -ll May 13 10:47:47 | set open fds limit to 20000000/20000000 May 13 10:47:47 | loading //lib/multipath/libchecktur.so checker May 13 10:47:47 | checker tur: message table size = 3 May 13 10:47:47 | loading //lib/multipath/libprioconst.so prioritizer May 13 10:47:47 | foreign library "nvme" loaded successfully May 13 10:47:47 | sda: size = 10485760 May 13 10:47:47 | sda: vendor = 3PARdata May 13 10:47:47 | sda: product = VV May 13 10:47:47 | sda: rev = 3313 May 13 10:47:47 | sda: h:b:t:l = 2:0:0:0 May 13 10:47:47 | sda: tgt_node_name = iqn.2000-05.com.3pardata:20210002ac023192 May 13 10:47:47 | sda: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sda: serial = CZ39074677 May 13 10:47:47 | sda: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sda: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sda: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sda: tur state = up May 13 10:47:47 | sdb: size = 10485760 May 13 10:47:47 | sdb: vendor = 3PARdata May 13 10:47:47 | sdb: product = VV May 13 10:47:47 | sdb: rev = 3313 May 13 10:47:47 | sdb: h:b:t:l = 2:0:0:1 May 13 10:47:47 | sdb: tgt_node_name = iqn.2000-05.com.3pardata:20210002ac023192 May 13 10:47:47 | sdb: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdb: serial = CZ39074677 May 13 10:47:47 | sdb: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdb: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdb: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdb: tur state = up May 13 10:47:47 | sdc: size = 10485760 May 13 10:47:47 | sdc: vendor = 3PARdata May 13 10:47:47 | sdc: product = VV May 13 10:47:47 | sdc: rev = 3313 May 13 10:47:47 | sdc: h:b:t:l = 3:0:0:0 May 13 10:47:47 | sdc: tgt_node_name = iqn.2000-05.com.3pardata:20220002ac023192 May 13 10:47:47 | sdc: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdc: serial = CZ39074677 May 13 10:47:47 | sdc: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdc: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdc: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdc: tur state = up May 13 10:47:47 | sdd: size = 10485760 May 13 10:47:47 | sdd: vendor = 3PARdata May 13 10:47:47 | sdd: product = VV May 13 10:47:47 | sdd: rev = 3313 May 13 10:47:47 | sdd: h:b:t:l = 3:0:0:1 May 13 10:47:47 | sdd: tgt_node_name = iqn.2000-05.com.3pardata:20220002ac023192 May 13 10:47:47 | sdd: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdd: serial = CZ39074677 May 13 10:47:47 | sdd: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdd: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdd: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdd: tur state = up May 13 10:47:47 | sde: size = 10485760 May 13 10:47:47 | sde: vendor = 3PARdata May 13 10:47:47 | sde: product = VV May 13 10:47:47 | sde: rev = 3313 May 13 10:47:47 | sde: h:b:t:l = 4:0:0:0 May 13 10:47:47 | sde: tgt_node_name = iqn.2000-05.com.3pardata:21210002ac023192 May 13 10:47:47 | sde: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sde: serial = CZ39074677 May 13 10:47:47 | sde: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sde: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sde: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sde: tur state = up May 13 10:47:47 | sdf: size = 10485760 May 13 10:47:47 | sdf: vendor = 3PARdata May 13 10:47:47 | sdf: product = VV May 13 10:47:47 | sdf: rev = 3313 May 13 10:47:47 | sdf: h:b:t:l = 4:0:0:1 May 13 10:47:47 | sdf: tgt_node_name = iqn.2000-05.com.3pardata:21210002ac023192 May 13 10:47:47 | sdf: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdf: serial = CZ39074677 May 13 10:47:47 | sdf: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdf: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdf: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdf: tur state = up May 13 10:47:47 | sdg: size = 10485760 May 13 10:47:47 | sdg: vendor = 3PARdata May 13 10:47:47 | sdg: product = VV May 13 10:47:47 | sdg: rev = 3313 May 13 10:47:47 | sdg: h:b:t:l = 5:0:0:0 May 13 10:47:47 | sdg: tgt_node_name = iqn.2000-05.com.3pardata:21220002ac023192 May 13 10:47:47 | sdg: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdg: serial = CZ39074677 May 13 10:47:47 | sdg: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdg: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdg: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdg: tur state = up May 13 10:47:47 | sdh: size = 10485760 May 13 10:47:47 | sdh: vendor = 3PARdata May 13 10:47:47 | sdh: product = VV May 13 10:47:47 | sdh: rev = 3313 May 13 10:47:47 | sdh: h:b:t:l = 5:0:0:1 May 13 10:47:47 | sdh: tgt_node_name = iqn.2000-05.com.3pardata:21220002ac023192 May 13 10:47:47 | sdh: 1018 cyl, 166 heads, 62 sectors/track, start at 0 May 13 10:47:47 | sdh: serial = CZ39074677 May 13 10:47:47 | sdh: detect_checker = yes (setting: multipath internal) May 13 10:47:47 | sdh: path_checker = tur (setting: storage device autodetected) May 13 10:47:47 | sdh: checker timeout = 30 s (setting: kernel sysfs) May 13 10:47:47 | sdh: tur state = up May 13 10:47:47 | loop0: device node name blacklisted May 13 10:47:47 | loop1: device node name blacklisted May 13 10:47:47 | loop2: device node name blacklisted May 13 10:47:47 | loop3: device node name blacklisted May 13 10:47:47 | loop4: device node name blacklisted May 13 10:47:47 | loop5: device node name blacklisted May 13 10:47:47 | loop6: device node name blacklisted May 13 10:47:47 | loop7: device node name blacklisted May 13 10:47:47 | dm-0: device node name blacklisted May 13 10:47:47 | dm-1: device node name blacklisted ===== paths list ===== uuid hcil dev dev_t pri dm_st chk_st vend/prod/rev dev_st 2:0:0:0 sda 8:0 -1 undef undef 3PARdata,VV unknown 2:0:0:1 sdb 8:16 -1 undef undef 3PARdata,VV unknown 3:0:0:0 sdc 8:32 -1 undef undef 3PARdata,VV unknown 3:0:0:1 sdd 8:48 -1 undef undef 3PARdata,VV unknown 4:0:0:0 sde 8:64 -1 undef undef 3PARdata,VV unknown 4:0:0:1 sdf 8:80 -1 undef undef 3PARdata,VV unknown 5:0:0:0 sdg 8:96 -1 undef undef 3PARdata,VV unknown 5:0:0:1 sdh 8:112 -1 undef undef 3PARdata,VV unknown May 13 10:47:47 | libdevmapper version 1.02.167 (2019-11-30) May 13 10:47:47 | DM multipath kernel driver v1.13.0 May 13 10:47:47 | params = 1 queue_if_no_path 1 alua 1 1 service-time 0 4 2 8:16 1 1 8:48 1 1 8:80 1 1 8:112 1 1 May 13 10:47:47 | status = 2 0 1 0 1 1 A 0 4 2 8:16 A 0 0 1 8:48 A 0 0 1 8:80 A 0 0 1 8:112 A 0 0 1 May 13 10:47:47 | sdb: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | loading //lib/multipath/libpriosysfs.so prioritizer May 13 10:47:47 | sdb: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdb: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdb: sysfs prio = 50 May 13 10:47:47 | sdd: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdd: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdd: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdd: sysfs prio = 50 May 13 10:47:47 | sdf: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdf: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdf: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdf: sysfs prio = 50 May 13 10:47:47 | sdh: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdh: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdh: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdh: sysfs prio = 50 mpathd (360002ac000000000000002d600023192) dm-1 3PARdata,VV size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 2:0:0:1 sdb 8:16 active ready running |- 3:0:0:1 sdd 8:48 active ready running |- 4:0:0:1 sdf 8:80 active ready running `- 5:0:0:1 sdh 8:112 active ready running May 13 10:47:47 | params = 1 queue_if_no_path 1 alua 1 1 service-time 0 4 2 8:0 1 1 8:32 1 1 8:64 1 1 8:96 1 1 May 13 10:47:47 | status = 2 0 1 0 1 1 A 0 4 2 8:0 A 0 0 1 8:32 A 0 0 1 8:64 A 0 0 1 8:96 A 0 0 1 May 13 10:47:47 | sda: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sda: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sda: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sda: sysfs prio = 50 May 13 10:47:47 | sdc: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdc: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdc: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdc: sysfs prio = 50 May 13 10:47:47 | sde: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sde: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sde: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sde: sysfs prio = 50 May 13 10:47:47 | sdg: detect_prio = yes (setting: multipath internal) May 13 10:47:47 | sdg: prio = sysfs (setting: storage device autodetected) May 13 10:47:47 | sdg: prio args = "" (setting: storage device autodetected) May 13 10:47:47 | sdg: sysfs prio = 50 mpathc (360002ac000000000000002d700023192) dm-0 3PARdata,VV size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=50 status=active |- 2:0:0:0 sda 8:0 active ready running |- 3:0:0:0 sdc 8:32 active ready running |- 4:0:0:0 sde 8:64 active ready running `- 5:0:0:0 sdg 8:96 active ready running May 13 10:47:47 | unloading sysfs prioritizer May 13 10:47:47 | unloading const prioritizer May 13 10:47:47 | unloading tur checker ```
```shell root@m20c20:/home/ep192# multipath -v3 -ll May 13 10:51:41 | set open fds limit to 20000000/20000000 May 13 10:51:41 | loading //lib/multipath/libchecktur.so checker May 13 10:51:41 | checker tur: message table size = 3 May 13 10:51:41 | loading //lib/multipath/libprioconst.so prioritizer May 13 10:51:41 | foreign library "nvme" loaded successfully May 13 10:51:41 | loop0: device node name blacklisted May 13 10:51:41 | loop1: device node name blacklisted May 13 10:51:41 | loop2: device node name blacklisted May 13 10:51:41 | loop3: device node name blacklisted May 13 10:51:41 | loop4: device node name blacklisted May 13 10:51:41 | loop5: device node name blacklisted May 13 10:51:41 | loop6: device node name blacklisted May 13 10:51:41 | loop7: device node name blacklisted May 13 10:51:41 | dm-1: device node name blacklisted ===== no paths ===== May 13 10:51:41 | libdevmapper version 1.02.167 (2019-11-30) May 13 10:51:41 | DM multipath kernel driver v1.13.0 May 13 10:51:41 | params = 1 queue_if_no_path 1 alua 0 0 May 13 10:51:41 | status = 2 0 0 0 0 0 mpathd (360002ac000000000000002d600023192) dm-1 ##,## size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw May 13 10:51:41 | unloading const prioritizer May 13 10:51:41 | unloading tur checker ```
@krakazyabra - I am unable to recreate this issue on CentOS and Kubernetes 1.19. I am using the HPE CSI Driver 1.4.0. Can you give me some more detail on the Kubernetes version? Host OS? What does your 3PAR multipath.conf look like?
Hello.
```shell root@m20c20:/home/ep192# cat /etc/multipath.conf defaults { user_friendly_names yes } devices { device { features 0 prio alua rr_min_io 100 path_grouping_policy group_by_prio hardware_handler "1 alua" failback immediate path_selector "round-robin 0" product "Server" fast_io_fail_tmo 5 path_checker tur dev_loss_tmo infinity rr_weight uniform vendor "Nimble" no_path_retry 18 } } ```
we're using kubernetes v1.20.5, OS is Ubuntu 20.04.1 LTS Also we use hpe-linux-config
```yaml kind: ConfigMap apiVersion: v1 metadata: name: hpe-linux-config namespace: hpe-storage data: config.json: |- [ { "category": "iscsi", "severity": "warning", "description": "Manual startup of iSCSI nodes on boot. Can be set in /etc/iscsi/iscsid.conf", "parameter": "startup", "recommendation": "manual" }, { "category": "iscsi", "severity": "warning", "description": "Replacement_timeout of 10 seconds is recommended for faster failover of I/O by multipath on path failures. Can be set in /etc/iscsi/iscsid.conf", "parameter": "replacement_timeout", "recommendation": "10" }, { "category": "iscsi", "severity": "warning", "description": "Minimum login timeout of 15 seconds is recommended with iSCSI. Can be set in /etc/iscsi/iscsid.conf", "parameter": "login_timeout", "recommendation": "15" }, { "category": "iscsi", "severity": "warning", "description": "Minimum timeout of 10 seconds is recommended with noop requests. Can be set in /etc/iscsi/iscsid.conf", "parameter": "noop_out_timeout", "recommendation": "10" }, { "category": "iscsi", "severity": "info", "description": "Minimum cmds_max of 512 is recommended for each session if handling multiple LUN's. Can be set in /etc/iscsi/iscsid.conf", "parameter": "cmds_max", "recommendation": "512" }, { "category": "iscsi", "severity": "warning", "description": "Minimum queue_depth of 256 is recommended for each iSCSI session/path. Can be set in /etc/iscsi/iscsid.conf", "parameter": "queue_depth", "recommendation": "256" }, { "category": "iscsi", "severity": "info", "description": "Minimum number of sessions per iSCSI login is recommended to be 1 by default. If additional sessions are needed this can be set in /etc/iscsi/iscsid.conf. If NCM is running, please change min_session_per_array in /etc/ncm.conf and restart nlt service instead", "parameter": "nr_sessions", "recommendation": "1" }, { "category": "multipath", "severity": "critical", "description": "product attribute recommended to be set to Server in /etc/multipath.conf", "parameter": "product", "recommendation": "\"Server\"" }, { "category": "multipath", "severity": "critical", "description": "alua prioritizer is recommended. Can be set in /etc/multipath.conf", "parameter": "prio", "recommendation": "alua" }, { "category": "multipath", "severity": "critical", "description": "scsi_dh_alua device handler is recommended. Can be set in /etc/multipath.conf", "parameter": "hardware_handler", "recommendation": "\"1 alua\"" }, { "category": "multipath", "severity": "warning", "description": "immediate failback setting is recommended. Can be set in /etc/multipath.conf", "parameter": "failback", "recommendation": "immediate" }, { "category": "multipath", "severity": "critical", "description": "immediately fail i/o on transient path failures to retry on other paths, value=1. Can be set in /etc/multipath.conf", "parameter": "fast_io_fail_tmo", "recommendation": "5" }, { "category": "multipath", "severity": "critical", "description": "queueing is recommended for 150 seconds, with no_path_retry value of 30. Can be set in /etc/multipath.conf", "parameter": "no_path_retry", "recommendation": "18" }, { "category": "multipath", "severity": "warning", "description": "service-time path selector is recommended. Can be set in /etc/multipath.conf", "parameter": "path_selector", "recommendation": "\"round-robin 0\"" }, { "category": "multipath", "severity": "critical", "description": "vendor attribute recommended to be set to Nimble in /etc/multipath.conf", "parameter": "vendor", "recommendation": "\"Nimble\"" }, { "category": "multipath", "severity": "critical", "description": "group paths according to ALUA path priority of active/standby. Recommended to be set to group_by_prio in /etc/multipath.conf", "parameter": "path_grouping_policy", "recommendation": "group_by_prio" }, { "category": "multipath", "severity": "critical", "description": "tur path checker is recommended. Can be set in /etc/multipath.conf", "parameter": "path_checker", "recommendation": "tur" }, { "category": "multipath", "severity": "critical", "description": "infinite value is recommended for timeout in cases of device loss for FC. Can be set in /etc/multipath.conf", "parameter": "dev_loss_tmo", "recommendation": "infinity" }, { "category": "multipath", "severity": "critical", "description": "fine tune 0", "parameter": "features", "recommendation": "0" }, { "category": "multipath", "severity": "critical", "description": "fine tune 1", "parameter": "rr_weight", "recommendation": "uniform" }, { "category": "multipath", "severity": "critical", "description": "fine tune 2", "parameter": "rr_min_io", "recommendation": "100" } ] ```
```yaml
# Configuration to deploy the HPE 3PAR Primera Storage CSP service
#
# example usage: kubectl create -f
```yaml
# Configuration to deploy the HPE CSI driver compatible with
# Kubernetes = v1.18
#
# example usage: kubectl create -f
apiVersion: storage.hpe.com/v1 kind: HPEReplicationDeviceInfo metadata: name: replicationdevices spec: target_array_details: - targetCpg: SSD_r6 targetSnapCpg: SSD_r6 targetName: 3par01 targetSecret: 3par01 targetSecretNamespace: hpe-storage ```yaml ```
```yaml
apiVersion: v1
stringData:
serviceName: primera3par-csp-svc
servicePort: "8080"
backend: <3PAR_IP>
username:
```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: 3par-hosting-1 provisioner: csi.hpe.com parameters: csi.storage.k8s.io/fstype: ext4 csi.storage.k8s.io/provisioner-secret-name: 3par01 csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage csi.storage.k8s.io/controller-publish-secret-name: 3par01 csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage csi.storage.k8s.io/node-stage-secret-name: 3par01 csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage csi.storage.k8s.io/node-publish-secret-name: 3par01 csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage # Uncomment for k8s 1.15 for resize support csi.storage.k8s.io/controller-expand-secret-name: 3par01 csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage # Use pre-existing CPG on 3PAR/Primera cpg: SSD_r6 provisioning_type: tpvv accessProtocol: iscsi allowVolumeExpansion: true reclaimPolicy: Delete ```
@sneharai4 @raunakkumar This appears to be isolated to how the 3PAR CSP handles volumes with Kubernetes running on Ubuntu nodes. My initial tests were using CentOS nodes and I was unable to recreate the issue. 3PAR/Primera multipathing is part of the multipath driver shipping with RHEL/CentOS.
Upon creating a new cluster using Ubuntu 20.04 with kubeadm 1.20.4 to match customer scenario, I was able to recreate the scenario. I am using the latest HPE CSI Driver 1.4.2 Helm chart.
Here is my output using the same PVCs and Pod YAML files from the customer:
root@kube-ubuntu:~# k get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
my-pvc-1 Bound pvc-05267728-fff2-45ec-a9ac-facc0ba3079b 5Gi RWO hpe-standard 34s
my-pvc-2 Bound pvc-13222920-f614-4749-935d-706751a4e462 5Gi RWO hpe-standard 3s
root@kube-ubuntu:~# multipath -ll
root@kube-ubuntu:~# cat /etc/multipath.conf
defaults {
user_friendly_names yes
}
devices {
device {
path_checker tur
hardware_handler "1 alua"
path_grouping_policy group_by_prio
no_path_retry 30
dev_loss_tmo infinity
prio alua
failback immediate
vendor "Nimble"
path_selector "service-time 0"
product "Server"
fast_io_fail_tmo 5
}
}
root@kube-ubuntu:~# k get pods
NAME READY STATUS RESTARTS AGE
my-pod-1 1/1 Running 0 100s
my-pod-2 1/1 Running 0 34s
root@kube-ubuntu:~# multipath -ll
mpathb (360002ac000000000000000dd0007ea94) dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 35:0:0:1 sdg 8:96 active ready running
|- 34:0:0:1 sdf 8:80 active ready running
`- 33:0:0:1 sde 8:64 active ready running
mpatha (360002ac000000000000000dc0007ea94) dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 33:0:0:0 sdb 8:16 active ready running
|- 35:0:0:0 sdd 8:48 active ready running
`- 34:0:0:0 sdc 8:32 active ready running
root@kube-ubuntu:~# k delete pod my-pod-2
pod "my-pod-2" deleted
root@kube-ubuntu:~# multipath -ll
mpatha (360002ac000000000000000dc0007ea94) dm-1 ##,##
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
root@kube-ubuntu:~# iscsiadm -m session
iscsiadm: No active sessions.
root@kube-ubuntu:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
my-pod-1 1/1 Running 0 21m
root@kube-ubuntu:~# kubectl logs my-pod-1
bin/sh: 1: cannot create /data/mydata.txt: Input/output error
bin/sh: 1: cannot create /data/mydata.txt: Input/output error
@krakazyabra Ubuntu OS is not supported by 3PAR/Primera. Please check the support matrix here, https://scod.hpedev.io/container_storage_provider/hpe_3par_primera/index.html#platform_requirements
@krakazyabra The issue is our conformance checks haven't applied the proper /etc/multipath.conf settings for Ubuntu and 3PAR. To work around this, you can manually configure the /etc/multipath.conf with the correct settings per HPE 3PAR Ubuntu Operating System Implementation Guide. Reboot the nodes and it should be fixed.
defaults {
polling_interval 10
max_fds 8192
}
devices {
device {
product "VV"
path_grouping_policy group_by_prio
features "0"
prio alua
no_path_retry 18
hardware_handler "1 alua"
failback immediate
rr_min_io 100
vendor "3PARdata"
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_selector "round-robin 0"
path_checker tur
checker tur
}
device {
prio alua
no_path_retry 30
path_selector "service-time 0"
path_checker tur
dev_loss_tmo infinity
failback immediate
fast_io_fail_tmo 5
path_grouping_policy group_by_prio
vendor "Nimble"
product "Server"
hardware_handler "1 alua"
}
}
Hello @c-snell
Thanks for this workaround. I've added it to my multipath.conf, multipath -t
now shows me this additions as well.
But problem still exists, when I'm deleting a pod, second exported pvc also disappearing.
```shell defaults { verbosity 2 polling_interval 10 max_polling_interval 40 reassign_maps "no" multipath_dir "//lib/multipath" path_selector "service-time 0" path_grouping_policy "failover" uid_attribute "ID_SERIAL" prio "const" prio_args "" features "0" path_checker "tur" alias_prefix "mpath" failback "manual" rr_min_io 1000 rr_min_io_rq 1 max_fds 8192 rr_weight "uniform" queue_without_daemon "no" flush_on_last_del "no" user_friendly_names "no" fast_io_fail_tmo 5 bindings_file "/etc/multipath/bindings" wwids_file "/etc/multipath/wwids" prkeys_file "/etc/multipath/prkeys" log_checker_err always all_tg_pt "no" retain_attached_hw_handler "yes" detect_prio "yes" detect_checker "yes" force_sync "no" strict_timing "no" deferred_remove "no" config_dir "/etc/multipath/conf.d" delay_watch_checks "no" delay_wait_checks "no" san_path_err_threshold "no" san_path_err_forget_rate "no" san_path_err_recovery_time "no" marginal_path_err_sample_time "no" marginal_path_err_rate_threshold "no" marginal_path_err_recheck_gap_time "no" marginal_path_double_failed_time "no" find_multipaths "on" uxsock_timeout 4000 retrigger_tries 0 retrigger_delay 10 missing_uev_wait_timeout 30 skip_kpartx "no" disable_changed_wwids ignored remove_retries 0 ghost_delay "no" find_multipaths_timeout -10 enable_foreign "" marginal_pathgroups "no" } blacklist { devnode "^drbd[0-9]+" devnode "^(ram|zram|raw|loop|fd|md|dm-|sr|scd|st|dcssblk)[0-9]" devnode "^(td|hd|vd)[a-z]" devnode "^cciss!c[0-9]d[0-9]*" device { vendor "SGI" product "Universal Xport" } device { vendor "^DGC" product "LUNZ" } device { vendor "EMC" product "LUNZ" } device { vendor "DELL" product "Universal Xport" } device { vendor "IBM" product "Universal Xport" } device { vendor "IBM" product "S/390" } device { vendor "LENOVO" product "Universal Xport" } device { vendor "(NETAPP|LSI|ENGENIO)" product "Universal Xport" } device { vendor "STK" product "Universal Xport" } device { vendor "SUN" product "Universal Xport" } device { vendor "(Intel|INTEL)" product "VTrak V-LUN" } device { vendor "Promise" product "VTrak V-LUN" } device { vendor "Promise" product "Vess V-LUN" } } blacklist_exceptions { property "(SCSI_IDENT_|ID_WWN)" } devices { device { vendor "NVME" product ".*" uid_attribute "ID_WWN" path_checker "none" retain_attached_hw_handler "no" } device { vendor "APPLE" product "Xserve RAID" path_grouping_policy "multibus" } device { vendor "DEC" product "HSG80" path_grouping_policy "group_by_prio" path_checker "hp_sw" hardware_handler "1 hp_sw" prio "hp_sw" no_path_retry "queue" } device { vendor "HP" product "A6189A" path_grouping_policy "multibus" no_path_retry 12 } device { vendor "(COMPAQ|HP)" product "(MSA|HSV)1[01]0" path_grouping_policy "group_by_prio" path_checker "hp_sw" hardware_handler "1 hp_sw" prio "hp_sw" no_path_retry 12 } device { vendor "(COMPAQ|HP)" product "MSA VOLUME" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "(COMPAQ|HP)" product "(HSV1[01]1|HSV2[01]0|HSV3[046]0|HSV4[05]0)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "HP" product "(MSA2[02]12fc|MSA2012i)" path_grouping_policy "multibus" no_path_retry 18 } device { vendor "HP" product "(MSA2012sa|MSA23(12|24)(fc|i|sa)|MSA2000s VOLUME)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "HP" product "MSA [12]0[45]0 SA[NS]" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "HP" product "HSVX700" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "HP" product "LOGICAL VOLUME" path_grouping_policy "multibus" no_path_retry 12 } device { vendor "HP" product "(P2000 G3 FC|P2000G3 FC/iSCSI|P2000 G3 SAS|P2000 G3 iSCSI)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "LEFTHAND" product "(P4000|iSCSIDisk|FCDISK)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 18 } device { vendor "SGI" product "TP9100" path_grouping_policy "multibus" } device { vendor "SGI" product "TP9[3457]00" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SGI" product "IS" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SGI" product "^DD[46]A-" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "DDN" product "SAN DataDirector" path_grouping_policy "multibus" } device { vendor "DDN" product "^EF3010" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "DDN" product "^(EF3015|S2A|SFA)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "NEXENTA" product "COMSTAR" path_grouping_policy "group_by_serial" no_path_retry 30 } device { vendor "TEGILE" product "(ZEBI-(FC|ISCSI)|INTELLIFLASH)" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 10 } device { vendor "EMC" product "SYMMETRIX" path_grouping_policy "multibus" no_path_retry 6 } device { vendor "^DGC" product "^(RAID|DISK|VRAID)" product_blacklist "LUNZ" path_grouping_policy "group_by_prio" path_checker "emc_clariion" hardware_handler "1 emc" prio "emc" failback "immediate" no_path_retry 60 } device { vendor "EMC" product "Invista" product_blacklist "LUNZ" path_grouping_policy "multibus" no_path_retry 5 } device { vendor "XtremIO" product "XtremApp" path_grouping_policy "multibus" } device { vendor "COMPELNT" product "Compellent Vol" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "DELL" product "^MD3" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "NVME" product "^EMC PowerMax_" path_grouping_policy "multibus" } device { vendor "FSC" product "CentricStor" path_grouping_policy "group_by_serial" } device { vendor "FUJITSU" product "ETERNUS_DX(H|L|M|400|8000)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 10 } device { vendor "(EUROLOGC|EuroLogc)" product "FC2502" path_grouping_policy "multibus" } device { vendor "FUJITSU" product "E[234]000" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 10 } device { vendor "FUJITSU" product "E[68]000" path_grouping_policy "multibus" no_path_retry 10 } device { vendor "(HITACHI|HP)" product "^OPEN-" path_grouping_policy "multibus" } device { vendor "HITACHI" product "^DF" path_grouping_policy "group_by_prio" prio "hds" failback "immediate" no_path_retry "queue" } device { vendor "HITACHI" product "^DF600F" path_grouping_policy "multibus" } device { vendor "IBM" product "ProFibre 4000R" path_grouping_policy "multibus" } device { vendor "IBM" product "^1722-600" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1724" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1726" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1742" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1746" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1813" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1814" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1815" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^1818" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^3526" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^(3542|3552)" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "IBM" product "^2105" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "IBM" product "^1750500" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "IBM" product "^2107900" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "IBM" product "^2145" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "IBM" product "S/390 DASD ECKD" product_blacklist "S/390" path_grouping_policy "multibus" uid_attribute "ID_UID" path_checker "directio" no_path_retry "queue" } device { vendor "IBM" product "S/390 DASD FBA" product_blacklist "S/390" path_grouping_policy "multibus" uid_attribute "ID_UID" path_checker "directio" no_path_retry "queue" } device { vendor "IBM" product "^IPR" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "IBM" product "1820N00" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "(XIV|IBM)" product "(NEXTRA|2810XIV)" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "(TMS|IBM)" product "(RamSan|FlashSystem)" path_grouping_policy "multibus" } device { vendor "IBM" product "^(DCS9900|2851)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "AIX" product "VDASD" path_grouping_policy "multibus" no_path_retry 60 } device { vendor "IBM" product "3303[ ]+NVDISK" no_path_retry 60 } device { vendor "AIX" product "NVDISK" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 60 } device { vendor "LENOVO" product "DE_Series" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "NETAPP" product "LUN" path_grouping_policy "group_by_prio" features "2 pg_init_retries 50" prio "ontap" failback "immediate" no_path_retry "queue" flush_on_last_del "yes" dev_loss_tmo "infinity" user_friendly_names "no" } device { vendor "(NETAPP|LSI|ENGENIO)" product "INF-01-00" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SolidFir" product "SSD SAN" path_grouping_policy "multibus" no_path_retry 24 } device { vendor "NVME" product "^NetApp ONTAP Controller" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "NEC" product "DISK ARRAY" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" } device { vendor "^Pillar" product "^Axiom" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" } device { vendor "^Oracle" product "^Oracle FS" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" } device { vendor "STK" product "BladeCtlr" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "STK" product "OPENstorage" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "STK" product "FLEXLINE 380" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "StorEdge 3" path_grouping_policy "multibus" } device { vendor "SUN" product "STK6580_6780" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "CSM[12]00_R" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "LCSM100_[IEFS]" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "SUN_6180" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "ArrayStorage" product_blacklist "Universal Xport" path_grouping_policy "group_by_prio" path_checker "rdac" features "2 pg_init_retries 50" hardware_handler "1 rdac" prio "rdac" failback "immediate" no_path_retry 30 } device { vendor "SUN" product "(Sun Storage|ZFS Storage|COMSTAR)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "PIVOT3" product "RAIGE VOLUME" path_grouping_policy "multibus" no_path_retry "queue" } device { vendor "(NexGen|Pivot3)" product "(TierStore|vSTAC)" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "(Intel|INTEL)" product "Multi-Flex" product_blacklist "VTrak V-LUN" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "(LIO-ORG|SUSE)" product "RBD" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 12 } device { vendor "DataCore" product "SANmelody" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "DataCore" product "Virtual Disk" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry "queue" } device { vendor "PURE" product "FlashArray" path_grouping_policy "multibus" } device { vendor "HUAWEI" product "XSG1" path_grouping_policy "group_by_prio" prio "alua" } device { vendor "KOVE" product "XPD" path_grouping_policy "multibus" } device { vendor "NFINIDAT" product "InfiniBox" path_grouping_policy "group_by_prio" path_selector "round-robin 0" prio "alua" failback 30 rr_weight "priorities" no_path_retry "fail" rr_min_io 1 rr_min_io_rq 1 flush_on_last_del "yes" fast_io_fail_tmo 15 dev_loss_tmo 15 } device { vendor "KMNRIO" product "K2" path_grouping_policy "multibus" } device { vendor "NEXSAN" product "NXS-B0" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 15 } device { vendor "NEXSAN" product "SATAB" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 15 } device { vendor "Nexsan" product "(NestOS|NST5000)" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "VIOLIN" product "SAN ARRAY$" path_grouping_policy "group_by_serial" no_path_retry 30 } device { vendor "VIOLIN" product "SAN ARRAY ALUA" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "VIOLIN" product "CONCERTO ARRAY" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "(XIOTECH|XIOtech)" product "ISE" path_grouping_policy "multibus" no_path_retry 12 } device { vendor "(XIOTECH|XIOtech)" product "IGLU DISK" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "(XIOTECH|XIOtech)" product "Magnitude" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "Promise" product "VTrak" product_blacklist "VTrak V-LUN" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "Promise" product "Vess" product_blacklist "Vess V-LUN" path_grouping_policy "group_by_prio" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "^IFT" product ".*" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "DotHill" product "SANnet" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "DotHill" product "R/Evo" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "DotHill" product "^DH" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" no_path_retry 30 } device { vendor "AStor" product "NeoSapphire" path_grouping_policy "multibus" no_path_retry 30 } device { vendor "INSPUR" product "MCS" path_grouping_policy "group_by_prio" prio "alua" failback "immediate" } device { vendor "3PARdata" product "VV" path_grouping_policy "group_by_prio" getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n" path_selector "round-robin 0" path_checker "tur" features "0" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 18 rr_min_io 100 fast_io_fail_tmo 10 dev_loss_tmo "infinity" } device { vendor "Nimble" product "Server" path_grouping_policy "group_by_prio" path_selector "service-time 0" path_checker "tur" hardware_handler "1 alua" prio "alua" failback "immediate" no_path_retry 30 fast_io_fail_tmo 5 dev_loss_tmo "infinity" } } overrides { } ```
@krakazyabra Hmm, I am not experiencing the volume disconnects after changing the /etc/multipath.conf
. I have a similar output to your multipath -t
.
Can you share a copy of the updated multipath.conf
? Did you reboot your nodes after updating the /etc/multipath.conf
?
Also have you opened a support case for this?
HPE Alletra 9000, Primera and 3PAR Hardware Contract Type | Phone Number |
---|---|
Warranty and Foundation Care | 800-633-3600 |
Proactive Care (PC) | 866-211-5211 |
Datacenter Care (DC) | 888-751-2149 |
Hello, @c-snell multipath.conf
root@m20c20:/home/ep192# cat /etc/multipath.conf
defaults {
polling_interval 10
max_fds 8192
}
devices {
device {
product "VV"
path_grouping_policy group_by_prio
features "0"
prio alua
no_path_retry 18
hardware_handler "1 alua"
failback immediate
rr_min_io 100
vendor "3PARdata"
getuid_callout "/lib/udev/scsi_id --whitelisted --device=/dev/%n"
path_selector "round-robin 0"
path_checker tur
checker tur
}
device {
prio alua
no_path_retry 30
path_selector "service-time 0"
path_checker tur
dev_loss_tmo infinity
failback immediate
fast_io_fail_tmo 5
path_grouping_policy group_by_prio
vendor "Nimble"
product "Server"
hardware_handler "1 alua"
}
}
No, I didn't make restart, only reload multipathd daemon. We're using pxe boot for all our servers, so this config should be included into our global pxe image. And this is not so fast)
Can confirm: after updating base PXE image and including multipath config there, problem still presents, even after reboot.
Hi! I've updated to 2.0.0 release. I was using same pods and pvc.
Here are multipath -ll -v 2
reports.
Create 2 pods with pvc:
root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f000023192 dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 2:0:0:2 sdc 8:32 active ready running
|- 3:0:0:2 sdg 8:96 active ready running
`- 4:0:0:2 sdk 8:160 active ready running
360002ac000000000000002f100023192 dm-3 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 5:0:0:3 sdp 8:240 active ready running
|- 2:0:0:3 sdd 8:48 active ready running
|- 3:0:0:3 sdh 8:112 active ready running
`- 4:0:0:3 sdl 8:176 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 5:0:0:1 sdn 8:208 active ready running
|- 2:0:0:1 sdb 8:16 active ready running
|- 3:0:0:1 sdf 8:80 active ready running
`- 4:0:0:1 sdj 8:144 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 5:0:0:0 sdm 8:192 active ready running
|- 2:0:0:0 sda 8:0 active ready running
|- 3:0:0:0 sde 8:64 active ready running
`- 4:0:0:0 sdi 8:128 active ready running
Delete pod my-pod-1:
root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f100023192 dm-3 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:3 sdl 8:176 active ready running
`- 5:0:0:3 sdp 8:240 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:1 sdj 8:144 active ready running
`- 5:0:0:1 sdn 8:208 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:0 sdi 8:128 active ready running
`- 5:0:0:0 sdm 8:192 active ready running
This is much better! At least, not all connections were unexport.
Create pod my-pod-1 back:
root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f000023192 dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:2 sdk 8:160 active ready running
|- 5:0:0:2 sdo 8:224 active ready running
|- 2:0:0:2 sdc 8:32 active ready running
`- 3:0:0:2 sdg 8:96 active ready running
360002ac000000000000002f100023192 dm-3 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:3 sdl 8:176 active ready running
|- 5:0:0:3 sdp 8:240 active ready running
|- 2:0:0:3 sdd 8:48 active ready running
`- 3:0:0:3 sdh 8:112 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:1 sdj 8:144 active ready running
|- 5:0:0:1 sdn 8:208 active ready running
|- 2:0:0:1 sdb 8:16 active ready running
`- 3:0:0:1 sdf 8:80 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 4:0:0:0 sdi 8:128 active ready running
|- 5:0:0:0 sdm 8:192 active ready running
|- 2:0:0:0 sda 8:0 active ready running
`- 3:0:0:0 sde 8:64 active ready running
And now delete my-pod-2
root@m20c20:/home/ep192# multipath -ll -v 2
360002ac000000000000002f000023192 dm-2 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 5:0:0:2 sdo 8:224 active ready running
360002ac000000000000002ee00023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 5:0:0:1 sdn 8:208 active ready running
360002ac000000000000002ef00023192 dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
`- 5:0:0:0 sdm 8:192 active ready running
I cannot say, that it's working perfect, because several times I saw, that all connections were unexported. But it's much better than in beginning.
@krakazyabra - Glad to hear it is working better. There have been a number of improvements done on the driver with regards to 3PAR/Primera. As of now the engineering team has not validated the HPE CSI Driver on Ubuntu nodes using HPE Alletra 9000, Primera, or 3PAR storage due to limited support of Ubuntu by the storage arrays. Here is the current support matrix: Platform Requirements. All validations have been done with RHEL, CentOS, or SLES. Full support for Ubuntu OS with HPE Primera and Alletra is slated for later this year and will then be validated on the HPE CSI Driver in an upcoming release.
I've made one more test, more complicated. We have another 3par (the same model). So, we connected it and deployed new storageClass, using the same csi-driver (just few new yamls)
replication.yaml
apiVersion: storage.hpe.com/v1
kind: HPEReplicationDeviceInfo
metadata:
name: replicationdevices-3par02
spec:
target_array_details:
- targetCpg: SSD_r6
targetSnapCpg: SSD_r6
targetName: 3par02
targetSecret: 3par02
targetSecretNamespace: hpe-storage
secrets.yaml
apiVersion: v1
stringData:
serviceName: primera3par-csp-svc
servicePort: "8080"
backend: 10.36.115.11
username: username
password: password
kind: Secret
metadata:
name: 3par02
namespace: hpe-storage
type: Opaque
storage_class.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: 3par02
provisioner: csi.hpe.com
parameters:
csi.storage.k8s.io/fstype: ext4
csi.storage.k8s.io/provisioner-secret-name: 3par02
csi.storage.k8s.io/provisioner-secret-namespace: hpe-storage
csi.storage.k8s.io/controller-publish-secret-name: 3par02
csi.storage.k8s.io/controller-publish-secret-namespace: hpe-storage
csi.storage.k8s.io/node-stage-secret-name: 3par02
csi.storage.k8s.io/node-stage-secret-namespace: hpe-storage
csi.storage.k8s.io/node-publish-secret-name: 3par02
csi.storage.k8s.io/node-publish-secret-namespace: hpe-storage
# Uncomment for k8s 1.15 for resize support
csi.storage.k8s.io/controller-expand-secret-name: 3par02
csi.storage.k8s.io/controller-expand-secret-namespace: hpe-storage
# Use pre-existing CPG on 3PAR/Primera
cpg: SSD_r6
provisioning_type: tpvv
accessProtocol: iscsi
# remoteCopyGroup: k8sRemoteCopyGroup
# replicationDevices: replicationdevices
allowVolumeExpansion: true
reclaimPolicy: Delete
And created 2 pods. my-pod-1 has pvc from storageClass 3par
, but second pod now has pvc from 3par02
storageClass.
Then I deleted one pod, as usual. And what do you think?
Csi-drivered worked as expected! It disconnect only volume from deleted node.
create 2 pods:
root@m20c20:/home/ep192/open-iscsi/iscsiuio# multipath -ll -v 2
360002ac0000000000000009900019d4a dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 2:0:0:0 sda 8:0 active ready running
|- 3:0:0:0 sdb 8:16 active ready running
|- 4:0:0:0 sdc 8:32 active ready running
`- 6:0:0:0 sdd 8:48 active ready running
360002ac000000000000002f200023192 dm-1 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 5:0:0:0 sde 8:64 active ready running
|- 7:0:0:0 sdf 8:80 active ready running
|- 8:0:0:0 sdg 8:96 active ready running
`- 9:0:0:0 sdh 8:112 active ready running
delete my-pod-1:
root@m20c20:/home/ep192/open-iscsi/iscsiuio# multipath -ll -v 2
360002ac0000000000000009900019d4a dm-0 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
|- 2:0:0:0 sda 8:0 active ready running
|- 3:0:0:0 sdb 8:16 active ready running
|- 4:0:0:0 sdc 8:32 active ready running
`- 6:0:0:0 sdd 8:48 active ready running
Run this test several times. And result always is the same.
IDK, but may be this info will help you somehow to speed up new release for Ubuntu ;-)
it seems we're using Groups Scoped Targets, but driver thinks that we're using Volume Scoped Targets.
source: https://newman.cloud/en/2019/05/28/sync-replication-and-peer-persistence-setup-hpe-nimble/
@kvaps tried to improve driver's logic and we found, that with this changes driver doesn't make logout from target after removing only one volume.
I performed a lot of testings and each test was good. Only one volume was deleting at once. All another were working well without any I/O errors.
some tests:
Create 2 pods with pvc (from messages below)
root@m20c2:/home/ep192# multipath -ll -v 2
mpathh (360002ac000000000000000c700019d4a) dm-11 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 3:0:0:0 sdf 8:80 active ready running
|- 4:0:0:0 sdg 8:96 active ready running
|- 5:0:0:0 sdh 8:112 active ready running
`- 2:0:0:0 sde 8:64 active ready running
mpathg (360002ac000000000000000c600019d4a) dm-10 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 3:0:0:1 sdb 8:16 active ready running
|- 4:0:0:1 sdc 8:32 active ready running
|- 5:0:0:1 sdd 8:48 active ready running
`- 2:0:0:1 sda 8:0 active ready running
Delete one pod
root@m20c2:/home/ep192# multipath -ll -v 2
mpathh (360002ac000000000000000c700019d4a) dm-11 3PARdata,VV
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=50 status=active
|- 3:0:0:0 sdf 8:80 active ready running
|- 4:0:0:0 sdg 8:96 active ready running
|- 5:0:0:0 sdh 8:112 active ready running
`- 2:0:0:0 sde 8:64 active ready running
and
root@m20c2:/home/ep192# iscsiadm -m session -P3
iSCSI Transport Class version 2.0-870
version 2.0-874
Target: iqn.2000-05.com.3pardata:20220002ac019d4a (non-flash)
Current Portal: 10.37.120.10:3260,1024
Persistent Portal: 10.37.120.10:3260,1024
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
Iface IPaddress: 10.37.131.151
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 10
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 10
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 16384
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 3 State: running
scsi3 Channel 00 Id 0 Lun: 0
Attached scsi disk sdf State: running
scsi3 Channel 00 Id 0 Lun: 254
Target: iqn.2000-05.com.3pardata:21210002ac019d4a (non-flash)
Current Portal: 10.37.120.11:3260,1025
Persistent Portal: 10.37.120.11:3260,1025
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
Iface IPaddress: 10.37.131.151
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 11
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 10
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 16384
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 4 State: running
scsi4 Channel 00 Id 0 Lun: 0
Attached scsi disk sdg State: running
scsi4 Channel 00 Id 0 Lun: 254
Target: iqn.2000-05.com.3pardata:21220002ac019d4a (non-flash)
Current Portal: 10.37.120.12:3260,1027
Persistent Portal: 10.37.120.12:3260,1027
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
Iface IPaddress: 10.37.131.151
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 12
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 10
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 16384
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 5 State: running
scsi5 Channel 00 Id 0 Lun: 0
Attached scsi disk sdh State: running
scsi5 Channel 00 Id 0 Lun: 254
Target: iqn.2000-05.com.3pardata:20210002ac019d4a (non-flash)
Current Portal: 10.37.120.9:3260,1026
Persistent Portal: 10.37.120.9:3260,1026
**********
Interface:
**********
Iface Name: default
Iface Transport: tcp
Iface Initiatorname: iqn.2006-07.org.company:dc1.m20c2
Iface IPaddress: 10.37.131.151
Iface HWaddress: <empty>
Iface Netdev: <empty>
SID: 9
iSCSI Connection State: LOGGED IN
iSCSI Session State: LOGGED_IN
Internal iscsid Session State: NO CHANGE
*********
Timeouts:
*********
Recovery Timeout: 10
Target Reset Timeout: 30
LUN Reset Timeout: 30
Abort Timeout: 15
*****
CHAP:
*****
username: <empty>
password: ********
username_in: <empty>
password_in: ********
************************
Negotiated iSCSI params:
************************
HeaderDigest: None
DataDigest: None
MaxRecvDataSegmentLength: 262144
MaxXmitDataSegmentLength: 65536
FirstBurstLength: 16384
MaxBurstLength: 262144
ImmediateData: Yes
InitialR2T: Yes
MaxOutstandingR2T: 1
************************
Attached SCSI devices:
************************
Host Number: 2 State: running
scsi2 Channel 00 Id 0 Lun: 0
Attached scsi disk sde State: running
scsi2 Channel 00 Id 0 Lun: 254
As you see, luns were deleted, but sessions to 3par are still alive. We think, that driver should perform logout operation only if there are no more active luns (exports).
@c-snell I am having exactly same issue on OpenShift 4.6 with v2.0.0 plugin version for 3Par so I don't think the problem is limited to Ubuntu only. I run multiple tests and often all iscsi sessions gets terminated once the PV is getting unmounted.
@c-snell We're facing exactly the same issue as @paka86 & @krakazyabra
OpenShift 4.6.44 / HPE CSI Driver 1.4.0
@paka86 @ondrejlinha try this version of driver (hpe-csi-driver):
docker pull ghcr.io/kvaps/hpestorage-csi-driver@sha256:c416404f6b3069edf1a409a1f637a43cf1a267c14866cc9173a784134cd37cb2
It has this changes. For us it completelly resolved all problems
@krakazyabra this has been fixed in this commit.
If there are more than one volume on one node, there is quite a good chance that when one of them is dissconnected (e.g. pod is removed), it will disconnect iscsi sessions used by other volumes. The result is disconected / read-only / broken fs of other pods running on the same node.
Here is normal state - 4 iscsi connections, used by all volumes:
And here is the state after one of the pod is killed. All iscsi connections were closed and all volumes are read only, possibly with broken filesystem and data loss: