[IMPROVEMENT] Support multipathd enabled environment

[innobead]

Is your improvement request related to a feature? Please describe (👍 if you like this request)

When running a multipath-enabled environment, users need to exclude Longhorn volumes manually, or else any volume will not be able to attach, because the dev will be managed by multipath. We need to make it automatically handled to provide a better user experience instead of struggling with unnecessary troubleshooting.

Describe the solution you'd like

• Have a node condition to understand if the node is unsatisfied with some requirements for readyness including multipath.
• Automatically update the multipath config to exclude longhorn volumes if it's installed, restart it to take effect if it's enabled and run. (best effort, can only do that if the system is managed by systemd) (ex: use vendor way for blacklist)

Describe alternatives you've considered

Manually do https://longhorn.io/kb/troubleshooting-volume-with-multipath/, but before that, users would encounter the attach problem and take time to figure out the cause.

Additional context

• https://www.suse.com/support/kb/doc/?id=000016563
• https://github.com/longhorn/longhorn/issues/5290#issuecomment-1730794090

cc @longhorn/dev

[doctorpangloss]

The underlying issue for many users of Longhorn is that some storage drivers using iSCSI enable multipathd

democratic-csi enable multipathd? Yes: https://github.com/democratic-csi/democratic-csi?tab=readme-ov-file#iscs

Also, you can see a related issue from AWS EBS CSI: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1417#issuecomment-1275246361

Seemingly the only thing that needs to be added to the blacklist based on that comment is:

      defaults {
        user_friendly_names yes
      }
      blacklist {
        device {
          vendor "IET"
          product "VIRTUAL-DISK"
        }
      }

because

$ ssh administrator@mynode udevadm info --query=all --name=/dev/sdd | grep -E "ID_MODEL=|ID_VENDOR="
E: ID_VENDOR=IET
E: ID_MODEL=VIRTUAL-DISK

this is the vendor and product name of the longhorn volume

[mantissahz]

Need to consider the node condition for multipathd, https://github.com/longhorn/longhorn-manager/pull/2763#discussion_r1671540945