search

NetApp / trident

Storage orchestrator for containers
Apache License 2.0
762 stars 222 forks source link

Trident `v23.01.0` ontap san works not reliable on centos7 nodes due to old multipath version #803

Closed khassel closed 1 year ago

khassel commented 1 year ago

Describe the bug

With v23.01.0 new code was introduced here which calls multipath -C /dev-dm....

We are using centos7 Nodes with multipath-tools v0.4.9 (05/33, 2016), this version has no -C switch.

This leads to problems when mounting an iscsi volume, here some logs of the trident pod:

~❯ kubectl -n trident logs trident-node-linux-vw4t2
Defaulted container "trident-main" out of: trident-main, driver-registrar
{"@timestamp":"2023-02-07T10:07:09Z","binary":"/trident_orchestrator","build_time":"Mon Jan 30 15:26:43 EST 2023","level":"info","message":"Running Trident storage orchestrator.","version":"23.01.0"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Initializing K8S helper frontend.","requestID":"f423519d-2dea-4ca5-84c0-f7ae197837b4","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Created Kubernetes clients.","namespace":"trident","version":"v1.23.8"}
{"@timestamp":"2023-02-07T10:07:09Z","gitVersion":"v1.23.8","level":"info","message":"K8S helper determined the container orchestrator version.","requestID":"f423519d-2dea-4ca5-84c0-f7ae197837b4","requestSource":"Internal","version":"1.23"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Initializing K8S helper frontend.","requestID":"71a98ea2-bebf-48c2-9076-5007371a3a37","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Added frontend.","name":"k8s_csi_node_helper"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Initializing CSI frontend.","name":"avk8st-node41","version":"23.01.0"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Force detach feature disabled.","requestID":"802036cc-f065-4bf1-8cca-2ff36ff2333e","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:09Z","capability":"STAGE_UNSTAGE_VOLUME","level":"info","message":"Enabling node service capability."}
{"@timestamp":"2023-02-07T10:07:09Z","capability":"EXPAND_VOLUME","level":"info","message":"Enabling node service capability."}
{"@timestamp":"2023-02-07T10:07:09Z","capability":"GET_VOLUME_STATS","level":"info","message":"Enabling node service capability."}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Enabling volume access mode.","mode":"SINGLE_NODE_WRITER"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Enabling volume access mode.","mode":"SINGLE_NODE_READER_ONLY"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Enabling volume access mode.","mode":"MULTI_NODE_READER_ONLY"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Enabling volume access mode.","mode":"MULTI_NODE_SINGLE_WRITER"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Enabling volume access mode.","mode":"MULTI_NODE_MULTI_WRITER"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Added frontend.","name":"csi"}
{"@timestamp":"2023-02-07T10:07:09Z","address":":17546","level":"info","message":"Initializing HTTPS REST frontend."}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Added frontend.","name":"HTTPS REST"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Added 0 existing volume(s)","requestID":"3001fdfa-b7cf-4d2e-9ef2-d9dfcb3c0290","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Trident bootstrapped successfully."}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Activating K8S helper frontend.","requestID":"8fb2275e-d0c4-451b-87f4-bd84bb639da3","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:09Z","address":":17546","level":"info","message":"Activating HTTPS REST frontend."}
{"@timestamp":"2023-02-07T10:07:09Z","level":"warning","message":"Could not determine determine published paths for volume: pvc-0e829154-0b8a-42b5-b74e-db3470fb9c0e"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"warning","message":"Could not determine determine published paths for volume: pvc-4eb6d724-48fc-4b16-92bd-b7649a970399"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"warning","message":"Could not determine determine published paths for volume: pvc-59c46ded-11fe-40a0-8dc7-8f74d0f30629"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Starting periodic node access reconciliation service.","requestID":"339653f8-fc42-4302-a92a-990cfef6bab3","requestSource":"Periodic"}
{"@timestamp":"2023-02-07T10:07:09Z","level":"info","message":"Activating CSI frontend.","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:10Z","IQN":"iqn.1994-05.com.redhat:c742e03b7393","level":"info","message":"Discovered iSCSI initiator name.","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:10Z","IP Addresses":"[10.200.2.1 10.50.149.233 172.27.73.10 172.27.75.11]","level":"info","message":"Discovered IP addresses.","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:10Z","error":"SMBActiveOnHost is not supported for linux","level":"warning","message":"Error discovering SMB service on host.","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:25Z","level":"info","message":"Updated Trident controller with node registration.","node":"avk8st-node41","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal"}
{"@timestamp":"2023-02-07T10:07:25Z","level":"info","message":"Listening for GRPC connections.","name":"/plugin/csi.sock","net":"unix"}
{"@timestamp":"2023-02-07T10:07:41Z","level":"warning","message":"Multipath device not found.","multipathDevice":"","requestID":"e784c863-0539-4dff-bda7-56ba419cd8ae","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:06Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:07Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:09Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:11Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:13Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:17Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:17Z","level":"error","message":"failed to unstage volume","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:17Z","level":"error","message":"GRPC error: rpc error: code = Internal desc = multipath device is not ready for flush","requestID":"6a68056b-5247-4b06-8d59-da05a151f999","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:18Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:19Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:20Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:22Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:25Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:29Z","device":"/dev/dm-8","error":"exit status 1","level":"error","message":"Flush pre-check failed for the device.","output":"Unknown switch: (null)\nmultipath-tools v0.4.9 (05/33, 2016)\nUsage:\n  multipath [-a|-A|-c|-w|-W] [-d] [-T tm:val] [-r] [-i] [-v lvl] [-p pol] [-b fil] [-q] [dev]\n  multipath -l|-ll|-f [-v lvl] [-b fil] [dev]\n  multipath -F [-v lvl]\n  multipath -t\n  multipath -h\n\nWhere:\n  -h      print this usage text\n  -l      show multipath topology (sysfs and DM info)\n  -ll     show multipath topology (maximum info)\n  -f      flush a multipath device map\n  -F      flush all multipath device maps\n  -a      add a device wwid to the wwids file\n  -A      add devices from kernel command line mpath.wwids\n          parameters to wwids file\n  -c      check if a device should be a path in a multipath device\n  -T tm:val\n          check if tm matches the multipathd timestamp. If so val is\n          whether or not the device is a path in a multipath device\n  -q      allow queue_if_no_path when multipathd is not running\n  -d      dry run, do not create or update devmaps\n  -t      dump internal hardware table\n  -r      force devmap reload\n  -i      ignore wwids file\n  -B      treat the bindings file as read only\n  -b fil  bindings file location\n  -w      remove a device from the wwids file\n  -W      reset the wwids file include only the current devices\n  -p pol  force all maps to specified path grouping policy :\n          . failover            one path per priority group\n          . multibus            all paths in one priority group\n          . group_by_serial     one priority group per serial\n          . group_by_prio       one priority group per priority lvl\n          . group_by_node_name  one priority group per target node\n  -v lvl  verbosity level\n          . 0 no output\n          . 1 print created devmap names only\n          . 2 default verbosity\n          . 3 print debug information\n  dev     action limited to:\n          . multipath named 'dev' (ex: mpath0) or\n          . multipath whose wwid is 'dev' (ex: 60051..)\n          . multipath including the path named 'dev' (ex: /dev/sda)\n          . multipath including the path with maj:min 'dev' (ex: 8:0)\n","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:29Z","level":"error","message":"failed to unstage volume","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:11:29Z","level":"error","message":"GRPC error: rpc error: code = Internal desc = multipath device is not ready for flush","requestID":"a92016b3-040d-45b1-bf3d-ddeae9e4db4a","requestSource":"CSI"}
{"@timestamp":"2023-02-07T10:12:25Z","CHAPInUse":"false","level":"warning","message":"Portal is missing LUN Number(s): [1].","portal":"172.27.75.209","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal","sessionInfoSource":"trackingInfo"}
{"@timestamp":"2023-02-07T10:12:25Z","CHAPInUse":"false","level":"warning","message":"Portal is missing LUN Number(s): [1].","portal":"172.27.73.109","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal","sessionInfoSource":"trackingInfo"}
{"@timestamp":"2023-02-07T10:12:25Z","CHAPInUse":"false","level":"warning","message":"Portal is missing LUN Number(s): [1].","portal":"172.27.73.209","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal","sessionInfoSource":"trackingInfo"}
{"@timestamp":"2023-02-07T10:12:25Z","CHAPInUse":"false","level":"warning","message":"Portal is missing LUN Number(s): [1].","portal":"172.27.75.109","requestID":"1180ca01-a626-4ec6-95ee-b48b911ff532","requestSource":"Internal","sessionInfoSource":"trackingInfo"}

We had to roll back to v22.

Environment

To Reproduce see description

Expected behavior trident v23 should work with centos7 Nodes

gnarl commented 1 year ago

Hi @khassel,

Unfortunately not even Centos 7.9 has picked up the "-C" switch for multipath. There is a workaround, that may work for you, where you can install device-mapper-multipath 0.8.4 from the EPEL-8 repository. Below are instructions where we were able to make this work in our lab environment. These installation steps may vary depending on your current Centos configuration. We strongly suggest that you test and verify this change before rolling out in a production environment. 

1. Install the following packages: 
    > sudo yum install -y lsscsi iscsi-initiator-utils sg3_utils

2. Create file /etc/yum.repos.d/test.repo with this content:
    > [EPEL-8]
    > baseurl = http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os
    > enabled = 1
    > gpgcheck = 0
    > name = EPEL8 RPMs
    > skip_if_unavailable = 1

3. Install device-mapper-multipath 0.8.4
    > sudo yum install device-mapper-multipath.x86_64

4. Verify that iscsi-initiator-utils version is 6.2.0.874-2.el7 or later:
    > rpm -q iscsi-initiator-utils

5. Set scanning to manual:
    > sudo sed -i 's/^\(node.session.scan\).*/\1 = manual/' /etc/iscsi/iscsid.conf

6. Enable multi pathing:
    > sudo mpathconf --enable --with_multipathd y --find_multipaths n

7. Ensure that iscsid and multipathd are running:
    > sudo systemctl enable --now iscsid multipathd
8. Enable and start iscsi:
    >  sudo systemctl enable --now iscsi
gnarl commented 1 year ago

Hi @khassel,

It was pointed out to me that the /etc/yum.repos.d/test.repo needs to be removed after installing device-mapper-multipath 0.8.4. This is so that the EPEL 8 repo doesn't conflict with future package management installs and updates.

khassel commented 1 year ago

Thank you for your replies @gnarl

There is a workaround, that may work for you ... We strongly suggest that you test and verify this change before rolling out in a production environment.

We decided to stay on the last v22 trident version until we have another node os running.

May you should highlight in the docs that running trident v23 with optap-san on centos7 nodes is not possible without the above workaround.

YvosOnTheHub commented 9 months ago

with Centos7, I also had to upgrade RPM before moving to the DMM update

the following did the trick (test on the NetApp Lab on Demand for Trident & Kubernetes)

rpm -q rpm
rpm-4.11.3-32.el7.x86_64

rpm -Uvh http://mirror.centos.org/centos/7/updates/x86_64/Packages/rpm-4.11.3-46.el7_9.x86_64.rpm \
https://rpmfind.net/linux/centos/7.9.2009/updates/x86_64/Packages/rpm-libs-4.11.3-46.el7_9.x86_64.rpm \
https://rpmfind.net/linux/centos/7.9.2009/updates/x86_64/Packages/rpm-python-4.11.3-46.el7_9.x86_64.rpm \
https://rpmfind.net/linux/centos/7.9.2009/updates/x86_64/Packages/rpm-build-4.11.3-46.el7_9.x86_64.rpm \
https://rpmfind.net/linux/centos/7.9.2009/updates/x86_64/Packages/rpm-build-libs-4.11.3-46.el7_9.x86_64.rpm \
https://rpmfind.net/linux/centos/7.9.2009/updates/x86_64/Packages/rpm-sign-4.11.3-46.el7_9.x86_64.rpm

rpm -q rpm
rpm-4.11.3-46.el7_9.x86_64